2024-09-13T05:04:25.138 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T05:04:25.154 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T05:04:25.249 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857 branch: reef description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_nbd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900857' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi no_nested_subset: false openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: reef ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: extra_packages: - rbd-nbd flavor: default sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: 7a853b1706d076d58e0da7a5d3b9b9eb36276880 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 248 sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 sleep_before_teardown: 0 subset: 116/128 suite: rbd suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7a853b1706d076d58e0da7a5d3b9b9eb36276880 targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJZmayn87lprQi+8OG7heq7Wj+WJEQypxXjSR2AqE6F5b4iR4OujIELR/3fcjG36j6n3aH5fNrDNlXtV8X4H7mU= smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAV69TgMtkN7Iao6DM43kZ2fCYq4JRdLAiOMezOg4x9420UWQYH4z4ccvxA7guUA7vJGffrD+TMeuGR4gEbq1LI= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBANUGUAfyNPu08llXue3k7KGxoL4/ERtW75hSIumivEzK5FuEea4KjfZdLSXXzDT/hoqAenSuBO8UbWmWRNHBY8= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/rbd-nbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_22:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T05:04:25.249 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa; will attempt to use it 2024-09-13T05:04:25.250 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa/tasks 2024-09-13T05:04:25.250 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T05:04:25.252 INFO:teuthology.task.internal:Checking packages... 2024-09-13T05:04:25.273 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '48a7de7e6e1701a88fd571c7e34908ac4b8ed412' 2024-09-13T05:04:25.273 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T05:04:25.273 INFO:teuthology.packaging:ref: None 2024-09-13T05:04:25.273 INFO:teuthology.packaging:tag: None 2024-09-13T05:04:25.273 INFO:teuthology.packaging:branch: reef 2024-09-13T05:04:25.273 INFO:teuthology.packaging:sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T05:04:25.273 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=reef 2024-09-13T05:04:25.547 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-843-g2c01458a-1jammy 2024-09-13T05:04:25.548 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T05:04:25.555 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T05:04:25.555 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T05:04:25.560 INFO:teuthology.task.internal:Saving configuration 2024-09-13T05:04:25.573 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T05:04:25.579 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T05:04:25.612 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 04:58:34.644264', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIzw6AShz4IzXNCHg1ltuTsrPIOLVphyzpM3VMj2bpnv'} 2024-09-13T05:04:25.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 04:58:34.645690', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfO8TjP9EpgtMG7zV2KijL2xjyRcVGVi7iLn5aTYSUa'} 2024-09-13T05:04:25.667 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 04:58:34.642602', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEYDGaZCbwx8sCfoVO0Piw/XCCED8Tq9gXd/GTD3+0oX'} 2024-09-13T05:04:25.667 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T05:04:25.673 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-13T05:04:25.674 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2024-09-13T05:04:25.674 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['client.0'] 2024-09-13T05:04:25.674 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T05:04:25.778 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fabaf475240>, signals=[15]) 2024-09-13T05:04:25.778 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T05:04:25.784 INFO:teuthology.task.internal:Opening connections... 2024-09-13T05:04:25.784 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-09-13T05:04:25.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:04:25.876 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:04:25.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:04:25.965 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T05:04:25.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:04:26.055 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T05:04:26.061 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-09-13T05:04:26.069 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-09-13T05:04:26.069 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Ubuntu" 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="22.04" 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_CODENAME=jammy 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:ID=ubuntu 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE=debian 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T05:04:26.118 INFO:teuthology.orchestra.run.smithi016.stdout:UBUNTU_CODENAME=jammy 2024-09-13T05:04:26.119 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-09-13T05:04:26.151 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-09-13T05:04:26.157 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-09-13T05:04:26.157 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-09-13T05:04:26.203 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T05:04:26.203 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="Ubuntu" 2024-09-13T05:04:26.203 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="22.04" 2024-09-13T05:04:26.203 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T05:04:26.204 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_CODENAME=jammy 2024-09-13T05:04:26.204 INFO:teuthology.orchestra.run.smithi079.stdout:ID=ubuntu 2024-09-13T05:04:26.204 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE=debian 2024-09-13T05:04:26.204 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T05:04:26.204 INFO:teuthology.orchestra.run.smithi079.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T05:04:26.204 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T05:04:26.204 INFO:teuthology.orchestra.run.smithi079.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T05:04:26.204 INFO:teuthology.orchestra.run.smithi079.stdout:UBUNTU_CODENAME=jammy 2024-09-13T05:04:26.204 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-09-13T05:04:26.234 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-09-13T05:04:26.240 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-09-13T05:04:26.240 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-09-13T05:04:26.287 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T05:04:26.287 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Ubuntu" 2024-09-13T05:04:26.287 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="22.04" 2024-09-13T05:04:26.287 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T05:04:26.287 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_CODENAME=jammy 2024-09-13T05:04:26.287 INFO:teuthology.orchestra.run.smithi122.stdout:ID=ubuntu 2024-09-13T05:04:26.288 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE=debian 2024-09-13T05:04:26.288 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T05:04:26.288 INFO:teuthology.orchestra.run.smithi122.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T05:04:26.288 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T05:04:26.288 INFO:teuthology.orchestra.run.smithi122.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T05:04:26.288 INFO:teuthology.orchestra.run.smithi122.stdout:UBUNTU_CODENAME=jammy 2024-09-13T05:04:26.288 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-09-13T05:04:26.309 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T05:04:26.318 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T05:04:26.328 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T05:04:26.328 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-09-13T05:04:26.330 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-09-13T05:04:26.333 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-09-13T05:04:26.337 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T05:04:26.343 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T05:04:26.343 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-09-13T05:04:26.379 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-09-13T05:04:26.381 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-09-13T05:04:26.401 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T05:04:26.497 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T05:04:26.511 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T05:04:26.511 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'client.0': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T05:04:26.511 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T05:04:26.512 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T05:04:26.512 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T05:04:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:04:26.517 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-09-13T05:04:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:04:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:04:26.519 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-13T05:04:26.519 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-13T05:04:26.564 INFO:teuthology.orchestra.run.smithi079.stdout:5.15.0-56-generic 2024-09-13T05:04:26.564 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.15.0-56-generic 2024-09-13T05:04:26.564 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get clean 2024-09-13T05:04:26.566 INFO:teuthology.orchestra.run.smithi016.stdout:5.15.0-56-generic 2024-09-13T05:04:26.566 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.15.0-56-generic 2024-09-13T05:04:26.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get clean 2024-09-13T05:04:26.567 INFO:teuthology.orchestra.run.smithi122.stdout:5.15.0-56-generic 2024-09-13T05:04:26.568 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.15.0-56-generic 2024-09-13T05:04:26.568 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get clean 2024-09-13T05:04:26.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2024-09-13T05:04:26.694 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get update 2024-09-13T05:04:26.698 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2024-09-13T05:04:26.831 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T05:04:26.846 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T05:04:26.846 INFO:teuthology.orchestra.run.smithi079.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T05:04:26.852 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T05:04:26.862 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T05:04:26.917 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T05:04:26.924 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T05:04:26.946 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T05:04:26.954 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T05:04:26.959 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T05:04:27.036 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T05:04:27.064 INFO:teuthology.orchestra.run.smithi079.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T05:04:27.071 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T05:04:27.088 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,022 kB] 2024-09-13T05:04:27.155 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,022 kB] 2024-09-13T05:04:27.171 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T05:04:27.175 INFO:teuthology.orchestra.run.smithi079.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T05:04:27.178 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T05:04:27.178 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T05:04:27.179 INFO:teuthology.orchestra.run.smithi079.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T05:04:27.189 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T05:04:27.191 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T05:04:27.194 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T05:04:27.197 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T05:04:27.197 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T05:04:27.198 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T05:04:27.198 INFO:teuthology.orchestra.run.smithi016.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T05:04:27.199 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T05:04:27.199 INFO:teuthology.orchestra.run.smithi079.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T05:04:27.203 INFO:teuthology.orchestra.run.smithi079.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T05:04:27.203 INFO:teuthology.orchestra.run.smithi079.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T05:04:27.210 INFO:teuthology.orchestra.run.smithi079.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T05:04:27.216 INFO:teuthology.orchestra.run.smithi079.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T05:04:27.216 INFO:teuthology.orchestra.run.smithi016.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T05:04:27.217 INFO:teuthology.orchestra.run.smithi079.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T05:04:27.217 INFO:teuthology.orchestra.run.smithi079.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T05:04:27.218 INFO:teuthology.orchestra.run.smithi079.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T05:04:27.218 INFO:teuthology.orchestra.run.smithi079.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T05:04:27.220 INFO:teuthology.orchestra.run.smithi016.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T05:04:27.220 INFO:teuthology.orchestra.run.smithi016.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T05:04:27.226 INFO:teuthology.orchestra.run.smithi016.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T05:04:27.227 INFO:teuthology.orchestra.run.smithi122.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T05:04:27.228 INFO:teuthology.orchestra.run.smithi122.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T05:04:27.232 INFO:teuthology.orchestra.run.smithi122.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T05:04:27.232 INFO:teuthology.orchestra.run.smithi122.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T05:04:27.234 INFO:teuthology.orchestra.run.smithi016.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T05:04:27.235 INFO:teuthology.orchestra.run.smithi016.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T05:04:27.236 INFO:teuthology.orchestra.run.smithi016.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T05:04:27.236 INFO:teuthology.orchestra.run.smithi016.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T05:04:27.237 INFO:teuthology.orchestra.run.smithi016.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T05:04:27.242 INFO:teuthology.orchestra.run.smithi122.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T05:04:27.251 INFO:teuthology.orchestra.run.smithi079.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T05:04:27.271 INFO:teuthology.orchestra.run.smithi016.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T05:04:27.271 INFO:teuthology.orchestra.run.smithi016.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T05:04:27.272 INFO:teuthology.orchestra.run.smithi016.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T05:04:27.272 INFO:teuthology.orchestra.run.smithi122.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T05:04:27.273 INFO:teuthology.orchestra.run.smithi016.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T05:04:27.273 INFO:teuthology.orchestra.run.smithi016.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T05:04:27.273 INFO:teuthology.orchestra.run.smithi016.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T05:04:27.274 INFO:teuthology.orchestra.run.smithi016.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T05:04:27.274 INFO:teuthology.orchestra.run.smithi016.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T05:04:27.275 INFO:teuthology.orchestra.run.smithi122.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T05:04:27.275 INFO:teuthology.orchestra.run.smithi122.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T05:04:27.276 INFO:teuthology.orchestra.run.smithi122.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T05:04:27.276 INFO:teuthology.orchestra.run.smithi016.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T05:04:27.277 INFO:teuthology.orchestra.run.smithi122.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T05:04:27.277 INFO:teuthology.orchestra.run.smithi122.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T05:04:27.277 INFO:teuthology.orchestra.run.smithi122.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T05:04:27.277 INFO:teuthology.orchestra.run.smithi122.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T05:04:27.278 INFO:teuthology.orchestra.run.smithi122.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T05:04:27.292 INFO:teuthology.orchestra.run.smithi122.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T05:04:27.308 INFO:teuthology.orchestra.run.smithi122.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T05:04:27.309 INFO:teuthology.orchestra.run.smithi122.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T05:04:27.309 INFO:teuthology.orchestra.run.smithi122.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T05:04:27.309 INFO:teuthology.orchestra.run.smithi122.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T05:04:27.392 INFO:teuthology.orchestra.run.smithi079.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T05:04:27.399 INFO:teuthology.orchestra.run.smithi122.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T05:04:27.426 INFO:teuthology.orchestra.run.smithi016.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T05:04:27.531 INFO:teuthology.orchestra.run.smithi079.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,022 kB] 2024-09-13T05:04:27.683 INFO:teuthology.orchestra.run.smithi122.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T05:04:27.847 INFO:teuthology.orchestra.run.smithi016.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T05:04:27.848 INFO:teuthology.orchestra.run.smithi079.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T05:04:27.856 INFO:teuthology.orchestra.run.smithi122.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T05:04:27.863 INFO:teuthology.orchestra.run.smithi016.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T05:04:27.869 INFO:teuthology.orchestra.run.smithi122.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T05:04:27.870 INFO:teuthology.orchestra.run.smithi122.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T05:04:27.872 INFO:teuthology.orchestra.run.smithi016.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T05:04:27.873 INFO:teuthology.orchestra.run.smithi016.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T05:04:27.884 INFO:teuthology.orchestra.run.smithi079.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T05:04:27.898 INFO:teuthology.orchestra.run.smithi079.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T05:04:27.899 INFO:teuthology.orchestra.run.smithi079.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T05:04:27.968 INFO:teuthology.orchestra.run.smithi079.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T05:04:27.969 INFO:teuthology.orchestra.run.smithi079.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T05:04:27.969 INFO:teuthology.orchestra.run.smithi016.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T05:04:27.971 INFO:teuthology.orchestra.run.smithi016.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T05:04:27.981 INFO:teuthology.orchestra.run.smithi016.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T05:04:27.981 INFO:teuthology.orchestra.run.smithi016.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T05:04:27.984 INFO:teuthology.orchestra.run.smithi079.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T05:04:27.984 INFO:teuthology.orchestra.run.smithi079.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T05:04:27.989 INFO:teuthology.orchestra.run.smithi122.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T05:04:27.990 INFO:teuthology.orchestra.run.smithi122.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T05:04:27.998 INFO:teuthology.orchestra.run.smithi016.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T05:04:28.006 INFO:teuthology.orchestra.run.smithi122.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T05:04:28.006 INFO:teuthology.orchestra.run.smithi122.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T05:04:28.006 INFO:teuthology.orchestra.run.smithi079.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T05:04:28.022 INFO:teuthology.orchestra.run.smithi122.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T05:04:28.027 INFO:teuthology.orchestra.run.smithi016.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T05:04:28.030 INFO:teuthology.orchestra.run.smithi016.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T05:04:28.031 INFO:teuthology.orchestra.run.smithi016.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T05:04:28.032 INFO:teuthology.orchestra.run.smithi016.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T05:04:28.037 INFO:teuthology.orchestra.run.smithi079.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T05:04:28.041 INFO:teuthology.orchestra.run.smithi122.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T05:04:28.042 INFO:teuthology.orchestra.run.smithi079.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T05:04:28.043 INFO:teuthology.orchestra.run.smithi079.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T05:04:28.044 INFO:teuthology.orchestra.run.smithi079.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T05:04:28.045 INFO:teuthology.orchestra.run.smithi122.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T05:04:28.049 INFO:teuthology.orchestra.run.smithi122.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T05:04:28.049 INFO:teuthology.orchestra.run.smithi122.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T05:04:28.093 INFO:teuthology.orchestra.run.smithi079.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T05:04:28.094 INFO:teuthology.orchestra.run.smithi079.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T05:04:28.094 INFO:teuthology.orchestra.run.smithi079.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T05:04:28.095 INFO:teuthology.orchestra.run.smithi079.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T05:04:28.097 INFO:teuthology.orchestra.run.smithi079.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T05:04:28.102 INFO:teuthology.orchestra.run.smithi122.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T05:04:28.102 INFO:teuthology.orchestra.run.smithi122.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T05:04:28.112 INFO:teuthology.orchestra.run.smithi016.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T05:04:28.112 INFO:teuthology.orchestra.run.smithi016.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T05:04:28.175 INFO:teuthology.orchestra.run.smithi079.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T05:04:28.176 INFO:teuthology.orchestra.run.smithi079.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T05:04:28.176 INFO:teuthology.orchestra.run.smithi079.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T05:04:28.177 INFO:teuthology.orchestra.run.smithi079.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T05:04:28.177 INFO:teuthology.orchestra.run.smithi079.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T05:04:33.514 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 16.0 MB in 2s (7,453 kB/s) 2024-09-13T05:04:33.517 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 16.0 MB in 2s (7,335 kB/s) 2024-09-13T05:04:33.874 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 16.0 MB in 2s (7,076 kB/s) 2024-09-13T05:04:34.947 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-09-13T05:04:34.965 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T05:04:34.995 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-09-13T05:04:35.014 DEBUG:teuthology.orchestra.run.smithi079:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T05:04:35.072 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-09-13T05:04:35.073 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-09-13T05:04:35.282 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-13T05:04:35.283 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2024-09-13T05:04:35.283 INFO:teuthology.orchestra.run.smithi079.stdout:Building dependency tree... 2024-09-13T05:04:35.283 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2024-09-13T05:04:35.284 INFO:teuthology.orchestra.run.smithi079.stdout:Reading state information... 2024-09-13T05:04:35.300 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T05:04:35.357 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-13T05:04:35.463 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T05:04:35.463 INFO:teuthology.orchestra.run.smithi079.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T05:04:35.463 INFO:teuthology.orchestra.run.smithi079.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T05:04:35.463 INFO:teuthology.orchestra.run.smithi079.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T05:04:35.463 INFO:teuthology.orchestra.run.smithi079.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T05:04:35.463 INFO:teuthology.orchestra.run.smithi079.stdout: python2.7-minimal 2024-09-13T05:04:35.464 INFO:teuthology.orchestra.run.smithi079.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T05:04:35.465 INFO:teuthology.orchestra.run.smithi079.stdout:The following additional packages will be installed: 2024-09-13T05:04:35.465 INFO:teuthology.orchestra.run.smithi079.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T05:04:35.465 INFO:teuthology.orchestra.run.smithi079.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T05:04:35.465 INFO:teuthology.orchestra.run.smithi079.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T05:04:35.467 INFO:teuthology.orchestra.run.smithi079.stdout:Suggested packages: 2024-09-13T05:04:35.467 INFO:teuthology.orchestra.run.smithi079.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T05:04:35.467 INFO:teuthology.orchestra.run.smithi079.stdout:Recommended packages: 2024-09-13T05:04:35.467 INFO:teuthology.orchestra.run.smithi079.stdout: thermald 2024-09-13T05:04:35.472 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T05:04:35.472 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T05:04:35.472 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T05:04:35.472 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T05:04:35.472 INFO:teuthology.orchestra.run.smithi122.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T05:04:35.473 INFO:teuthology.orchestra.run.smithi122.stdout: python2.7-minimal 2024-09-13T05:04:35.473 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T05:04:35.474 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2024-09-13T05:04:35.474 INFO:teuthology.orchestra.run.smithi122.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T05:04:35.474 INFO:teuthology.orchestra.run.smithi122.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T05:04:35.475 INFO:teuthology.orchestra.run.smithi122.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T05:04:35.476 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2024-09-13T05:04:35.476 INFO:teuthology.orchestra.run.smithi122.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T05:04:35.476 INFO:teuthology.orchestra.run.smithi122.stdout:Recommended packages: 2024-09-13T05:04:35.476 INFO:teuthology.orchestra.run.smithi122.stdout: thermald 2024-09-13T05:04:35.518 INFO:teuthology.orchestra.run.smithi079.stdout:The following NEW packages will be installed: 2024-09-13T05:04:35.519 INFO:teuthology.orchestra.run.smithi079.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T05:04:35.519 INFO:teuthology.orchestra.run.smithi079.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T05:04:35.519 INFO:teuthology.orchestra.run.smithi079.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T05:04:35.520 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages will be upgraded: 2024-09-13T05:04:35.521 INFO:teuthology.orchestra.run.smithi079.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T05:04:35.523 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2024-09-13T05:04:35.523 INFO:teuthology.orchestra.run.smithi122.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T05:04:35.523 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T05:04:35.523 INFO:teuthology.orchestra.run.smithi122.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T05:04:35.525 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages will be upgraded: 2024-09-13T05:04:35.525 INFO:teuthology.orchestra.run.smithi122.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T05:04:35.546 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2024-09-13T05:04:35.547 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2024-09-13T05:04:35.571 INFO:teuthology.orchestra.run.smithi079.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T05:04:35.571 INFO:teuthology.orchestra.run.smithi079.stdout:Need to get 113 MB of archives. 2024-09-13T05:04:35.571 INFO:teuthology.orchestra.run.smithi079.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T05:04:35.571 INFO:teuthology.orchestra.run.smithi079.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T05:04:35.577 INFO:teuthology.orchestra.run.smithi122.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T05:04:35.577 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 113 MB of archives. 2024-09-13T05:04:35.577 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T05:04:35.578 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T05:04:35.720 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T05:04:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T05:04:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T05:04:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T05:04:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T05:04:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: python2.7-minimal 2024-09-13T05:04:35.721 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T05:04:35.722 INFO:teuthology.orchestra.run.smithi016.stdout:The following additional packages will be installed: 2024-09-13T05:04:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T05:04:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T05:04:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T05:04:35.724 INFO:teuthology.orchestra.run.smithi016.stdout:Suggested packages: 2024-09-13T05:04:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T05:04:35.724 INFO:teuthology.orchestra.run.smithi016.stdout:Recommended packages: 2024-09-13T05:04:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: thermald 2024-09-13T05:04:35.766 INFO:teuthology.orchestra.run.smithi016.stdout:The following NEW packages will be installed: 2024-09-13T05:04:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T05:04:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T05:04:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T05:04:35.768 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages will be upgraded: 2024-09-13T05:04:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T05:04:35.819 INFO:teuthology.orchestra.run.smithi016.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T05:04:35.819 INFO:teuthology.orchestra.run.smithi016.stdout:Need to get 113 MB of archives. 2024-09-13T05:04:35.819 INFO:teuthology.orchestra.run.smithi016.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T05:04:35.819 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T05:04:36.026 INFO:teuthology.orchestra.run.smithi079.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T05:04:36.057 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T05:04:36.117 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T05:04:36.155 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T05:04:36.227 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T05:04:36.324 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T05:04:36.962 INFO:teuthology.orchestra.run.smithi079.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T05:04:36.962 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T05:04:36.962 INFO:teuthology.orchestra.run.smithi079.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T05:04:36.993 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T05:04:36.993 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T05:04:36.993 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T05:04:37.081 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T05:04:37.115 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T05:04:37.136 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T05:04:37.182 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T05:04:37.182 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T05:04:37.182 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T05:04:37.183 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T05:04:37.312 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T05:04:37.334 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T05:04:37.559 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 113 MB in 2s (70.6 MB/s) 2024-09-13T05:04:37.600 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 113 MB in 2s (68.4 MB/s) 2024-09-13T05:04:37.742 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T05:04:37.794 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T05:04:37.815 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 113 MB in 2s (68.9 MB/s) 2024-09-13T05:04:37.985 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T05:04:39.891 INFO:teuthology.orchestra.run.smithi079.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T05:04:39.895 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:39.919 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T05:04:39.923 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:39.967 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T05:04:39.971 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:40.016 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:40.190 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:40.199 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:42.167 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T05:04:42.188 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:42.265 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:42.417 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T05:04:42.434 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T05:04:42.437 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:42.455 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:42.516 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:42.547 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:42.762 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T05:04:42.783 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:42.820 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:42.979 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T05:04:42.999 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:43.011 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T05:04:43.028 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:43.032 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:43.070 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:49.598 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T05:04:49.703 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T05:04:49.731 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T05:04:49.830 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T05:04:49.831 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T05:04:49.964 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T05:04:50.006 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T05:04:50.118 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T05:04:50.147 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T05:04:50.267 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T05:04:50.306 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T05:04:50.429 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T05:04:50.435 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T05:04:50.452 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T05:04:50.495 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T05:04:50.579 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T05:04:50.601 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T05:04:50.645 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T05:04:50.729 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T05:04:50.751 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T05:04:50.796 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T05:04:56.082 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T05:04:56.119 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:56.157 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:56.296 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T05:04:56.301 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T05:04:56.333 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:56.338 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T05:04:56.371 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:56.376 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:58.998 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T05:04:59.001 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T05:04:59.130 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T05:04:59.135 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T05:04:59.471 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T05:04:59.478 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T05:04:59.593 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:59.606 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:59.663 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T05:04:59.716 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T05:04:59.736 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T05:04:59.806 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T05:04:59.844 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:04:59.863 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:00.150 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T05:05:00.278 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:00.429 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T05:05:00.557 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:01.173 INFO:teuthology.orchestra.run.smithi122.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T05:05:01.174 INFO:teuthology.orchestra.run.smithi122.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T05:05:01.174 INFO:teuthology.orchestra.run.smithi122.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T05:05:01.174 INFO:teuthology.orchestra.run.smithi122.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:01.192 INFO:teuthology.orchestra.run.smithi016.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T05:05:01.193 INFO:teuthology.orchestra.run.smithi016.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T05:05:01.193 INFO:teuthology.orchestra.run.smithi016.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T05:05:01.193 INFO:teuthology.orchestra.run.smithi016.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:01.334 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:01.355 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:01.906 INFO:teuthology.orchestra.run.smithi079.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T05:05:01.906 INFO:teuthology.orchestra.run.smithi079.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T05:05:01.906 INFO:teuthology.orchestra.run.smithi079.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T05:05:01.907 INFO:teuthology.orchestra.run.smithi079.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:02.060 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:03.198 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:03.385 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:03.694 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:04.608 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T05:05:04.734 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T05:05:04.845 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T05:05:04.860 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:04.935 INFO:teuthology.orchestra.run.smithi016.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T05:05:04.936 INFO:teuthology.orchestra.run.smithi016.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:04.971 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T05:05:05.070 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T05:05:05.097 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:05.164 INFO:teuthology.orchestra.run.smithi122.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T05:05:05.164 INFO:teuthology.orchestra.run.smithi122.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:05.196 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T05:05:05.314 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T05:05:05.370 INFO:teuthology.orchestra.run.smithi079.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T05:05:05.429 INFO:teuthology.orchestra.run.smithi079.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:19.756 INFO:teuthology.orchestra.run.smithi079.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T05:05:19.756 INFO:teuthology.orchestra.run.smithi079.stdout:Sourcing file `/etc/default/grub' 2024-09-13T05:05:19.788 INFO:teuthology.orchestra.run.smithi079.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T05:05:19.802 INFO:teuthology.orchestra.run.smithi122.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T05:05:19.802 INFO:teuthology.orchestra.run.smithi122.stdout:Sourcing file `/etc/default/grub' 2024-09-13T05:05:19.810 INFO:teuthology.orchestra.run.smithi122.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T05:05:19.811 INFO:teuthology.orchestra.run.smithi079.stdout:Generating grub configuration file ... 2024-09-13T05:05:19.834 INFO:teuthology.orchestra.run.smithi122.stdout:Generating grub configuration file ... 2024-09-13T05:05:19.894 INFO:teuthology.orchestra.run.smithi016.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T05:05:19.894 INFO:teuthology.orchestra.run.smithi016.stdout:Sourcing file `/etc/default/grub' 2024-09-13T05:05:19.905 INFO:teuthology.orchestra.run.smithi016.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T05:05:19.926 INFO:teuthology.orchestra.run.smithi016.stdout:Generating grub configuration file ... 2024-09-13T05:05:20.075 INFO:teuthology.orchestra.run.smithi079.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T05:05:20.096 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T05:05:20.103 INFO:teuthology.orchestra.run.smithi079.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:20.126 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:20.199 INFO:teuthology.orchestra.run.smithi016.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T05:05:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:20.430 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T05:05:20.436 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T05:05:20.452 INFO:teuthology.orchestra.run.smithi079.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T05:05:20.457 INFO:teuthology.orchestra.run.smithi079.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T05:05:20.505 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T05:05:20.511 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T05:05:20.524 INFO:teuthology.orchestra.run.smithi079.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T05:05:20.524 INFO:teuthology.orchestra.run.smithi016.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T05:05:20.529 INFO:teuthology.orchestra.run.smithi079.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T05:05:20.531 INFO:teuthology.orchestra.run.smithi016.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T05:05:20.609 INFO:teuthology.orchestra.run.smithi016.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T05:05:20.615 INFO:teuthology.orchestra.run.smithi016.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T05:05:20.665 INFO:teuthology.orchestra.run.smithi122.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T05:05:20.665 INFO:teuthology.orchestra.run.smithi122.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T05:05:20.665 INFO:teuthology.orchestra.run.smithi122.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T05:05:20.669 INFO:teuthology.orchestra.run.smithi079.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T05:05:20.669 INFO:teuthology.orchestra.run.smithi079.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T05:05:20.669 INFO:teuthology.orchestra.run.smithi079.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T05:05:20.698 INFO:teuthology.orchestra.run.smithi122.stdout:done 2024-09-13T05:05:20.702 INFO:teuthology.orchestra.run.smithi079.stdout:done 2024-09-13T05:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T05:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T05:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T05:05:20.832 INFO:teuthology.orchestra.run.smithi016.stdout:done 2024-09-13T05:05:21.187 DEBUG:teuthology.orchestra.run.smithi079:> dpkg -s linux-image-generic 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Package: linux-image-generic 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Status: install ok installed 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Priority: optional 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Section: kernel 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Installed-Size: 21 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: amd64 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Source: linux-meta 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Version: 5.15.0.121.121 2024-09-13T05:05:21.211 INFO:teuthology.orchestra.run.smithi079.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T05:05:21.212 INFO:teuthology.orchestra.run.smithi079.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T05:05:21.212 INFO:teuthology.orchestra.run.smithi079.stdout:Recommends: thermald 2024-09-13T05:05:21.212 INFO:teuthology.orchestra.run.smithi079.stdout:Description: Generic Linux kernel image 2024-09-13T05:05:21.212 INFO:teuthology.orchestra.run.smithi079.stdout: This package will always depend on the latest generic kernel image 2024-09-13T05:05:21.212 INFO:teuthology.orchestra.run.smithi079.stdout: available. 2024-09-13T05:05:21.212 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T05:05:21.212 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T05:05:21.212 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T05:05:21.213 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T05:05:21.213 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T05:05:21.213 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get clean 2024-09-13T05:05:21.222 DEBUG:teuthology.orchestra.run.smithi122:> dpkg -s linux-image-generic 2024-09-13T05:05:21.238 INFO:teuthology.orchestra.run.smithi122.stdout:Package: linux-image-generic 2024-09-13T05:05:21.238 INFO:teuthology.orchestra.run.smithi122.stdout:Status: install ok installed 2024-09-13T05:05:21.238 INFO:teuthology.orchestra.run.smithi122.stdout:Priority: optional 2024-09-13T05:05:21.238 INFO:teuthology.orchestra.run.smithi122.stdout:Section: kernel 2024-09-13T05:05:21.238 INFO:teuthology.orchestra.run.smithi122.stdout:Installed-Size: 21 2024-09-13T05:05:21.238 INFO:teuthology.orchestra.run.smithi122.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T05:05:21.239 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: amd64 2024-09-13T05:05:21.239 INFO:teuthology.orchestra.run.smithi122.stdout:Source: linux-meta 2024-09-13T05:05:21.239 INFO:teuthology.orchestra.run.smithi122.stdout:Version: 5.15.0.121.121 2024-09-13T05:05:21.239 INFO:teuthology.orchestra.run.smithi122.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T05:05:21.239 INFO:teuthology.orchestra.run.smithi122.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T05:05:21.239 INFO:teuthology.orchestra.run.smithi122.stdout:Recommends: thermald 2024-09-13T05:05:21.239 INFO:teuthology.orchestra.run.smithi122.stdout:Description: Generic Linux kernel image 2024-09-13T05:05:21.239 INFO:teuthology.orchestra.run.smithi122.stdout: This package will always depend on the latest generic kernel image 2024-09-13T05:05:21.239 INFO:teuthology.orchestra.run.smithi122.stdout: available. 2024-09-13T05:05:21.239 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T05:05:21.239 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T05:05:21.239 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T05:05:21.239 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-13T05:05:21.240 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T05:05:21.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get clean 2024-09-13T05:05:21.303 DEBUG:teuthology.orchestra.run.smithi016:> dpkg -s linux-image-generic 2024-09-13T05:05:21.332 INFO:teuthology.orchestra.run.smithi016.stdout:Package: linux-image-generic 2024-09-13T05:05:21.332 INFO:teuthology.orchestra.run.smithi016.stdout:Status: install ok installed 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Priority: optional 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Section: kernel 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Installed-Size: 21 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: amd64 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Source: linux-meta 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Version: 5.15.0.121.121 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Recommends: thermald 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Description: Generic Linux kernel image 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: This package will always depend on the latest generic kernel image 2024-09-13T05:05:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: available. 2024-09-13T05:05:21.334 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T05:05:21.334 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T05:05:21.334 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T05:05:21.334 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T05:05:21.334 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T05:05:21.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get clean 2024-09-13T05:05:21.412 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get update 2024-09-13T05:05:21.429 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2024-09-13T05:05:21.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2024-09-13T05:05:21.544 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T05:05:21.554 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T05:05:21.555 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T05:05:21.564 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T05:05:21.571 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T05:05:21.659 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T05:05:21.659 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T05:05:21.668 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T05:05:21.669 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T05:05:21.686 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T05:05:21.764 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T05:05:21.827 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T05:05:22.945 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-09-13T05:05:22.963 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T05:05:23.016 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-09-13T05:05:23.024 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-09-13T05:05:23.041 DEBUG:teuthology.orchestra.run.smithi079:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T05:05:23.091 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-13T05:05:23.093 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-09-13T05:05:23.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T05:05:23.174 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-13T05:05:23.209 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2024-09-13T05:05:23.209 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2024-09-13T05:05:23.280 INFO:teuthology.orchestra.run.smithi079.stdout:Building dependency tree... 2024-09-13T05:05:23.280 INFO:teuthology.orchestra.run.smithi079.stdout:Reading state information... 2024-09-13T05:05:23.382 INFO:teuthology.orchestra.run.smithi122.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T05:05:23.382 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T05:05:23.382 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T05:05:23.382 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T05:05:23.382 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T05:05:23.383 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T05:05:23.383 INFO:teuthology.orchestra.run.smithi122.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T05:05:23.383 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T05:05:23.385 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2024-09-13T05:05:23.385 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2024-09-13T05:05:23.423 INFO:teuthology.orchestra.run.smithi122.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T05:05:23.425 DEBUG:teuthology.orchestra.run.smithi122:> dpkg -s linux-image-generic 2024-09-13T05:05:23.446 INFO:teuthology.orchestra.run.smithi079.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T05:05:23.446 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T05:05:23.446 INFO:teuthology.orchestra.run.smithi079.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T05:05:23.446 INFO:teuthology.orchestra.run.smithi079.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T05:05:23.447 INFO:teuthology.orchestra.run.smithi079.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T05:05:23.447 INFO:teuthology.orchestra.run.smithi079.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T05:05:23.447 INFO:teuthology.orchestra.run.smithi079.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T05:05:23.447 INFO:teuthology.orchestra.run.smithi079.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Package: linux-image-generic 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Status: install ok installed 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Priority: optional 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Section: kernel 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Installed-Size: 21 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: amd64 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Source: linux-meta 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Version: 5.15.0.121.121 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T05:05:23.481 INFO:teuthology.orchestra.run.smithi122.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T05:05:23.482 INFO:teuthology.orchestra.run.smithi122.stdout:Recommends: thermald 2024-09-13T05:05:23.482 INFO:teuthology.orchestra.run.smithi122.stdout:Description: Generic Linux kernel image 2024-09-13T05:05:23.482 INFO:teuthology.orchestra.run.smithi122.stdout: This package will always depend on the latest generic kernel image 2024-09-13T05:05:23.482 INFO:teuthology.orchestra.run.smithi122.stdout: available. 2024-09-13T05:05:23.482 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-09-13T05:05:23.485 INFO:teuthology.orchestra.run.smithi079.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T05:05:23.487 DEBUG:teuthology.orchestra.run.smithi079:> dpkg -s linux-image-generic 2024-09-13T05:05:23.526 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.BdPHl34dXU 2024-09-13T05:05:23.527 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub/grub.cfg /tmp/tmp.BdPHl34dXU 2024-09-13T05:05:23.544 INFO:teuthology.orchestra.run.smithi079.stdout:Package: linux-image-generic 2024-09-13T05:05:23.544 INFO:teuthology.orchestra.run.smithi079.stdout:Status: install ok installed 2024-09-13T05:05:23.544 INFO:teuthology.orchestra.run.smithi079.stdout:Priority: optional 2024-09-13T05:05:23.544 INFO:teuthology.orchestra.run.smithi079.stdout:Section: kernel 2024-09-13T05:05:23.544 INFO:teuthology.orchestra.run.smithi079.stdout:Installed-Size: 21 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: amd64 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout:Source: linux-meta 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout:Version: 5.15.0.121.121 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout:Recommends: thermald 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout:Description: Generic Linux kernel image 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout: This package will always depend on the latest generic kernel image 2024-09-13T05:05:23.545 INFO:teuthology.orchestra.run.smithi079.stdout: available. 2024-09-13T05:05:23.546 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-09-13T05:05:23.582 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.BdPHl34dXU 2024-09-13T05:05:23.583 INFO:teuthology.orchestra.run.smithi016.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T05:05:23.583 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T05:05:23.583 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T05:05:23.583 INFO:teuthology.orchestra.run.smithi016.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T05:05:23.584 INFO:teuthology.orchestra.run.smithi016.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T05:05:23.584 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T05:05:23.584 INFO:teuthology.orchestra.run.smithi016.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T05:05:23.585 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T05:05:23.594 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.jHwYjNkJnF 2024-09-13T05:05:23.594 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub/grub.cfg /tmp/tmp.jHwYjNkJnF 2024-09-13T05:05:23.628 INFO:teuthology.orchestra.run.smithi016.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T05:05:23.630 DEBUG:teuthology.orchestra.run.smithi016:> dpkg -s linux-image-generic 2024-09-13T05:05:23.646 INFO:teuthology.orchestra.run.smithi016.stdout:Package: linux-image-generic 2024-09-13T05:05:23.646 INFO:teuthology.orchestra.run.smithi016.stdout:Status: install ok installed 2024-09-13T05:05:23.646 INFO:teuthology.orchestra.run.smithi016.stdout:Priority: optional 2024-09-13T05:05:23.646 INFO:teuthology.orchestra.run.smithi016.stdout:Section: kernel 2024-09-13T05:05:23.646 INFO:teuthology.orchestra.run.smithi016.stdout:Installed-Size: 21 2024-09-13T05:05:23.646 INFO:teuthology.orchestra.run.smithi016.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T05:05:23.646 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: amd64 2024-09-13T05:05:23.647 INFO:teuthology.orchestra.run.smithi016.stdout:Source: linux-meta 2024-09-13T05:05:23.647 INFO:teuthology.orchestra.run.smithi016.stdout:Version: 5.15.0.121.121 2024-09-13T05:05:23.647 INFO:teuthology.orchestra.run.smithi016.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T05:05:23.647 INFO:teuthology.orchestra.run.smithi016.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T05:05:23.647 INFO:teuthology.orchestra.run.smithi016.stdout:Recommends: thermald 2024-09-13T05:05:23.647 INFO:teuthology.orchestra.run.smithi016.stdout:Description: Generic Linux kernel image 2024-09-13T05:05:23.647 INFO:teuthology.orchestra.run.smithi016.stdout: This package will always depend on the latest generic kernel image 2024-09-13T05:05:23.647 INFO:teuthology.orchestra.run.smithi016.stdout: available. 2024-09-13T05:05:23.647 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-13T05:05:23.650 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.jHwYjNkJnF 2024-09-13T05:05:23.693 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.BymwZyNlaa 2024-09-13T05:05:23.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub/grub.cfg /tmp/tmp.BymwZyNlaa 2024-09-13T05:05:23.711 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.BdPHl34dXU is 10KB 2024-09-13T05:05:23.723 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.BdPHl34dXU 2024-09-13T05:05:23.728 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T05:05:23.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.BymwZyNlaa 2024-09-13T05:05:23.761 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.jHwYjNkJnF is 10KB 2024-09-13T05:05:23.774 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.jHwYjNkJnF 2024-09-13T05:05:23.780 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T05:05:23.782 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:05:23.782 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T05:05:23.782 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T05:05:23.834 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:05:23.834 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T05:05:23.834 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T05:05:23.854 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T05:05:23.854 DEBUG:teuthology.orchestra.run.smithi122:> sudo update-grub 2024-09-13T05:05:23.881 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.BymwZyNlaa is 10KB 2024-09-13T05:05:23.895 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.BymwZyNlaa 2024-09-13T05:05:23.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T05:05:23.902 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T05:05:23.902 DEBUG:teuthology.orchestra.run.smithi079:> sudo update-grub 2024-09-13T05:05:23.952 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:05:23.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T05:05:23.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T05:05:24.020 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T05:05:24.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo update-grub 2024-09-13T05:05:25.147 INFO:teuthology.orchestra.run.smithi079.stderr:Sourcing file `/etc/default/grub' 2024-09-13T05:05:25.149 INFO:teuthology.orchestra.run.smithi079.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T05:05:25.151 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-09-13T05:05:25.171 INFO:teuthology.orchestra.run.smithi122.stderr:Sourcing file `/etc/default/grub' 2024-09-13T05:05:25.173 INFO:teuthology.orchestra.run.smithi122.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T05:05:25.183 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-09-13T05:05:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub' 2024-09-13T05:05:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T05:05:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-09-13T05:05:25.323 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T05:05:25.329 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:25.370 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T05:05:25.376 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:25.496 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T05:05:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T05:05:25.569 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T05:05:25.574 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T05:05:25.612 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T05:05:25.616 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T05:05:25.617 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T05:05:25.623 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T05:05:25.668 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T05:05:25.673 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T05:05:25.723 INFO:teuthology.orchestra.run.smithi079.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T05:05:25.723 INFO:teuthology.orchestra.run.smithi079.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T05:05:25.723 INFO:teuthology.orchestra.run.smithi079.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T05:05:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T05:05:25.734 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T05:05:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-09-13T05:05:25.755 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-09-13T05:05:25.777 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T05:05:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T05:05:25.818 INFO:teuthology.orchestra.run.smithi122.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T05:05:25.818 INFO:teuthology.orchestra.run.smithi122.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T05:05:25.819 INFO:teuthology.orchestra.run.smithi122.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T05:05:25.829 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-09-13T05:05:25.870 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-09-13T05:05:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T05:05:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T05:05:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T05:05:25.953 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-09-13T05:05:25.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-09-13T05:05:55.785 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T05:05:55.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-13T05:05:55.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:05:55.878 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-13T05:05:55.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T05:05:55.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:05:55.958 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T05:05:55.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-13T05:05:55.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:06:30.337 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T05:06:30.589 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-13T05:06:39.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T05:06:39.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:06:39.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-13T05:06:39.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:06:55.807 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T05:06:57.729 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T05:07:04.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-13T05:07:04.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:07:05.853 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-09-13T05:07:09.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T05:07:09.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:07:10.444 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-13T05:07:11.326 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T05:07:11.326 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.15.0-121-generic"... 2024-09-13T05:07:11.326 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-13T05:07:11.377 INFO:teuthology.orchestra.run.smithi122.stdout:5.15.0-121-generic 2024-09-13T05:07:11.377 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T05:07:11.377 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T05:07:11.377 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T05:07:12.378 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-09-13T05:07:12.379 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T05:07:12.474 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-09-13T05:07:12.498 DEBUG:teuthology.parallel:result is None 2024-09-13T05:07:17.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-13T05:07:17.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:07:18.070 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-09-13T05:07:18.617 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-13T05:07:18.617 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-13T05:07:18.617 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-09-13T05:07:18.664 INFO:teuthology.orchestra.run.smithi079.stdout:5.15.0-121-generic 2024-09-13T05:07:18.664 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T05:07:18.664 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T05:07:18.664 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T05:07:19.665 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T05:07:19.665 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T05:07:19.735 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-09-13T05:07:19.761 DEBUG:teuthology.parallel:result is None 2024-09-13T05:07:39.618 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T05:07:51.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-13T05:07:51.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:07:51.843 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-13T05:07:52.458 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-13T05:07:52.458 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-13T05:07:52.458 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-13T05:07:52.507 INFO:teuthology.orchestra.run.smithi016.stdout:5.15.0-121-generic 2024-09-13T05:07:52.507 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T05:07:52.507 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T05:07:52.507 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T05:07:53.508 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T05:07:53.508 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T05:07:53.568 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-09-13T05:07:53.589 DEBUG:teuthology.parallel:result is None 2024-09-13T05:07:53.589 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T05:07:53.596 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T05:07:53.596 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T05:07:53.599 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T05:07:53.601 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T05:07:53.608 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T05:07:53.615 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T05:07:53.623 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T05:07:53.623 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T05:07:53.648 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T05:07:53.650 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T05:07:53.748 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T05:07:53.755 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T05:07:53.755 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T05:07:53.757 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T05:07:53.760 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T05:07:53.772 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T05:07:53.781 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T05:07:53.782 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T05:07:53.792 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T05:07:53.801 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T05:07:53.812 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T05:07:53.813 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T05:07:53.820 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T05:07:53.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T05:07:53.828 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T05:07:53.836 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T05:07:53.868 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T05:07:53.879 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T05:07:53.879 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T05:07:53.884 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T05:07:53.893 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T05:07:53.917 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T05:07:53.939 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T05:07:53.988 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:07:53.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T05:07:54.043 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T05:07:54.049 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T05:07:54.096 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:07:54.097 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T05:07:54.163 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T05:07:54.169 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T05:07:54.217 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:07:54.217 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T05:07:54.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-09-13T05:07:54.283 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-09-13T05:07:54.286 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-09-13T05:07:54.358 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T05:07:54.366 INFO:teuthology.task.internal:Starting timer... 2024-09-13T05:07:54.366 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T05:07:54.387 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T05:07:54.396 DEBUG:teuthology.task.selinux:Excluding smithi016: OS 'ubuntu' does not support SELinux 2024-09-13T05:07:54.396 DEBUG:teuthology.task.selinux:Excluding smithi079: OS 'ubuntu' does not support SELinux 2024-09-13T05:07:54.396 DEBUG:teuthology.task.selinux:Excluding smithi122: OS 'ubuntu' does not support SELinux 2024-09-13T05:07:54.397 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T05:07:54.397 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-13T05:07:54.397 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T05:07:54.397 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T05:07:54.406 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T05:07:54.418 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T05:07:54.419 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi079.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T05:15:32.505 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-09-13T05:15:32.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-13T05:15:32.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:15:32.617 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-13T05:15:32.697 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-13T05:15:32.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-13T05:15:32.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:15:32.807 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-09-13T05:15:32.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-13T05:15:32.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T05:15:32.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T05:15:32.995 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-13T05:15:33.076 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T05:15:33.078 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T05:15:33.091 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T05:15:33.091 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T05:15:33.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T05:15:33.094 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T05:15:33.095 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T05:15:33.097 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T05:15:33.097 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T05:15:33.122 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: Command line: ntpd -gq 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: ---------------------------------------------------- 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: corporation. Support and training for ntp-4 are 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: available at https://www.nwtime.org/support 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: ---------------------------------------------------- 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: proto: precision = 0.066 usec (-24) 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: basedate set to 2022-02-04 2024-09-13T05:15:33.123 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: gps base set to 2022-02-06 (week 2196) 2024-09-13T05:15:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T05:15:33.124 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T05:15:33.124 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T05:15:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T05:15:33.126 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T05:15:33.126 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T05:15:33.126 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T05:15:33.126 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: Listen normally on 3 ens1f0 172.21.15.16:123 2024-09-13T05:15:33.126 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: Listen normally on 4 lo [::1]:123 2024-09-13T05:15:33.126 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1564%4]:123 2024-09-13T05:15:33.126 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:33 ntpd[18409]: Listening on routing socket on fd #22 for interface updates 2024-09-13T05:15:33.127 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T05:15:33.127 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: Command line: ntpd -gq 2024-09-13T05:15:33.127 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: ---------------------------------------------------- 2024-09-13T05:15:33.127 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T05:15:33.127 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T05:15:33.127 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: corporation. Support and training for ntp-4 are 2024-09-13T05:15:33.127 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: available at https://www.nwtime.org/support 2024-09-13T05:15:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: ---------------------------------------------------- 2024-09-13T05:15:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: proto: precision = 0.043 usec (-24) 2024-09-13T05:15:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: basedate set to 2022-02-04 2024-09-13T05:15:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: gps base set to 2022-02-06 (week 2196) 2024-09-13T05:15:33.128 INFO:teuthology.orchestra.run.smithi079.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T05:15:33.129 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T05:15:33.129 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T05:15:33.129 INFO:teuthology.orchestra.run.smithi079.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T05:15:33.130 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T05:15:33.130 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T05:15:33.130 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T05:15:33.131 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: Listen normally on 3 enp3s0f1 172.21.15.79:123 2024-09-13T05:15:33.131 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: Listen normally on 4 lo [::1]:123 2024-09-13T05:15:33.131 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce87%5]:123 2024-09-13T05:15:33.131 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:33 ntpd[18506]: Listening on routing socket on fd #22 for interface updates 2024-09-13T05:15:33.147 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T05:15:33.147 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: Command line: ntpd -gq 2024-09-13T05:15:33.148 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: ---------------------------------------------------- 2024-09-13T05:15:33.148 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T05:15:33.148 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T05:15:33.148 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: corporation. Support and training for ntp-4 are 2024-09-13T05:15:33.148 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: available at https://www.nwtime.org/support 2024-09-13T05:15:33.148 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: ---------------------------------------------------- 2024-09-13T05:15:33.149 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: proto: precision = 0.047 usec (-24) 2024-09-13T05:15:33.149 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: basedate set to 2022-02-04 2024-09-13T05:15:33.149 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: gps base set to 2022-02-06 (week 2196) 2024-09-13T05:15:33.150 INFO:teuthology.orchestra.run.smithi122.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T05:15:33.150 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T05:15:33.150 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T05:15:33.151 INFO:teuthology.orchestra.run.smithi122.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T05:15:33.152 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T05:15:33.152 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T05:15:33.152 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T05:15:33.152 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: Listen normally on 3 enp3s0f1 172.21.15.122:123 2024-09-13T05:15:33.153 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: Listen normally on 4 lo [::1]:123 2024-09-13T05:15:33.153 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d515%5]:123 2024-09-13T05:15:33.153 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:33 ntpd[18513]: Listening on routing socket on fd #22 for interface updates 2024-09-13T05:15:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:34 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:34.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:34 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:34.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:34 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:34.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:34 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:34.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:34 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:34.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:34 ntpd[18506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:34.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:34 ntpd[18506]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:35 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:35 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:35 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:35 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:35 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:35 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:35.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:35 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:35.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:35 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:35.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:35 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:35.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:35 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:35.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:35 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:35.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:35 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:35.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:35 ntpd[18506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:35.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:35 ntpd[18506]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:36 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:36.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:36 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:36.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:36 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:36.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:36 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:36.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:36 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:36.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:36 ntpd[18506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:36.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:36 ntpd[18506]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:37.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:37 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:37.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:37 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:37.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:37 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:37.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:37 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:37.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:37 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:37.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:37 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:37.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:37 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:37.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:37 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:37.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:37 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:37.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:37 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:37.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:37 ntpd[18506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:37.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:37 ntpd[18506]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:37.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:37 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:37.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:37 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:37.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:37 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:37.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:37 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:37.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:37 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:37.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:37 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:38 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:38.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.125 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:38.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:38.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:38.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:38 ntpd[18506]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:38.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:38.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:38.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:38.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:38.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:38.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:38 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:39 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:39 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:39 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:39 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:39 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:39 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:39.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:39 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:39.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:39 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:39.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:39 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:39.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:39 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:39.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:39 ntpd[18506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:39.124 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:39 ntpd[18506]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:39.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:39 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:39.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:39 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:39.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:39 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:39.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:39 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:39.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:39 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:39.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:39 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:40.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.123 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:40.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:40.124 INFO:teuthology.orchestra.run.smithi016.stdout:13 Sep 05:15:40 ntpd[18409]: ntpd: time slew -0.000226 s 2024-09-13T05:15:40.124 INFO:teuthology.orchestra.run.smithi016.stdout:ntpd: time slew -0.000226s 2024-09-13T05:15:40.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T05:15:40.124 INFO:teuthology.orchestra.run.smithi016.stderr:13 Sep 05:15:40 ntpd[18409]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T05:15:40.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.126 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:40.127 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:40.127 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.127 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.127 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:40.127 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:40.127 INFO:teuthology.orchestra.run.smithi079.stdout:13 Sep 05:15:40 ntpd[18506]: ntpd: time slew -0.000440 s 2024-09-13T05:15:40.127 INFO:teuthology.orchestra.run.smithi079.stdout:ntpd: time slew -0.000440s 2024-09-13T05:15:40.128 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T05:15:40.128 INFO:teuthology.orchestra.run.smithi079.stderr:13 Sep 05:15:40 ntpd[18506]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T05:15:40.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:40.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:40.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:40.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:40.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:40.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:40 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T05:15:40.175 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== 2024-09-13T05:15:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:40.177 INFO:teuthology.orchestra.run.smithi079.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T05:15:40.177 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================== 2024-09-13T05:15:40.178 INFO:teuthology.orchestra.run.smithi079.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:40.178 INFO:teuthology.orchestra.run.smithi079.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:40.178 INFO:teuthology.orchestra.run.smithi079.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:40.178 INFO:teuthology.orchestra.run.smithi079.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:41.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:41 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:41.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:41 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:41.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:41 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:41.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:41 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:41.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:41 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:41.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:41 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:42.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:42.150 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:42 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:43.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:43 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:43.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:43 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:43.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:43 ntpd[18513]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T05:15:43.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:43 ntpd[18513]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T05:15:43.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:43 ntpd[18513]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T05:15:43.148 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:43 ntpd[18513]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T05:15:43.149 INFO:teuthology.orchestra.run.smithi122.stdout:13 Sep 05:15:43 ntpd[18513]: ntpd: time slew -0.000278 s 2024-09-13T05:15:43.149 INFO:teuthology.orchestra.run.smithi122.stdout:ntpd: time slew -0.000278s 2024-09-13T05:15:43.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:43 ntpd[18513]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T05:15:43.149 INFO:teuthology.orchestra.run.smithi122.stderr:13 Sep 05:15:43 ntpd[18513]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T05:15:43.203 INFO:teuthology.orchestra.run.smithi122.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T05:15:43.203 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================== 2024-09-13T05:15:43.203 INFO:teuthology.orchestra.run.smithi122.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:43.203 INFO:teuthology.orchestra.run.smithi122.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:43.203 INFO:teuthology.orchestra.run.smithi122.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:43.203 INFO:teuthology.orchestra.run.smithi122.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:15:43.204 INFO:teuthology.run_tasks:Running task install... 2024-09-13T05:15:43.212 DEBUG:teuthology.task.install:project ceph 2024-09-13T05:15:43.212 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}} 2024-09-13T05:15:43.212 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'} 2024-09-13T05:15:43.212 INFO:teuthology.task.install:Using flavor: default 2024-09-13T05:15:43.218 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T05:15:43.218 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2024-09-13T05:15:43.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-key list | grep Ceph 2024-09-13T05:15:43.219 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-key list | grep Ceph 2024-09-13T05:15:43.220 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-key list | grep Ceph 2024-09-13T05:15:43.284 INFO:teuthology.orchestra.run.smithi079.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T05:15:43.285 INFO:teuthology.orchestra.run.smithi016.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T05:15:43.311 INFO:teuthology.orchestra.run.smithi122.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T05:15:43.312 INFO:teuthology.orchestra.run.smithi079.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T05:15:43.312 INFO:teuthology.orchestra.run.smithi079.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T05:15:43.313 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-13T05:15:43.313 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T05:15:43.327 INFO:teuthology.orchestra.run.smithi016.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T05:15:43.327 INFO:teuthology.orchestra.run.smithi016.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T05:15:43.327 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-13T05:15:43.327 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T05:15:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T05:15:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T05:15:43.339 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-13T05:15:43.339 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T05:15:43.483 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default/ 2024-09-13T05:15:43.483 INFO:teuthology.task.install.deb:Package version is 18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:15:43.498 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default/ 2024-09-13T05:15:43.499 INFO:teuthology.task.install.deb:Package version is 18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:15:43.500 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default/ 2024-09-13T05:15:43.500 INFO:teuthology.task.install.deb:Package version is 18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:15:43.613 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:15:43.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T05:15:43.627 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:15:43.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T05:15:43.630 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get update 2024-09-13T05:15:43.630 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:15:43.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T05:15:43.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2024-09-13T05:15:43.645 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2024-09-13T05:15:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T05:15:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T05:15:43.812 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T05:15:43.812 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T05:15:43.815 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T05:15:43.825 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T05:15:43.826 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T05:15:43.826 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T05:15:43.842 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T05:15:43.843 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T05:15:43.843 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T05:15:43.851 INFO:teuthology.orchestra.run.smithi079.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T05:15:43.864 INFO:teuthology.orchestra.run.smithi122.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T05:15:43.868 INFO:teuthology.orchestra.run.smithi016.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T05:15:43.872 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-13T05:15:43.885 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-13T05:15:43.891 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-13T05:15:43.894 INFO:teuthology.orchestra.run.smithi079.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T05:15:43.907 INFO:teuthology.orchestra.run.smithi122.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T05:15:43.913 INFO:teuthology.orchestra.run.smithi016.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T05:15:43.915 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,421 B] 2024-09-13T05:15:43.921 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T05:15:43.927 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2024-09-13T05:15:43.934 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2024-09-13T05:15:43.942 INFO:teuthology.orchestra.run.smithi079.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2024-09-13T05:15:43.955 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,421 B] 2024-09-13T05:15:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,421 B] 2024-09-13T05:15:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 28.8 kB in 0s (84.4 kB/s) 2024-09-13T05:15:44.175 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 28.8 kB in 0s (81.9 kB/s) 2024-09-13T05:15:44.180 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 28.8 kB in 0s (78.5 kB/s) 2024-09-13T05:15:45.155 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-09-13T05:15:45.171 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-820-g48a7de7e-1jammy cephadm=18.2.4-820-g48a7de7e-1jammy ceph-mds=18.2.4-820-g48a7de7e-1jammy ceph-mgr=18.2.4-820-g48a7de7e-1jammy ceph-common=18.2.4-820-g48a7de7e-1jammy ceph-fuse=18.2.4-820-g48a7de7e-1jammy ceph-test=18.2.4-820-g48a7de7e-1jammy ceph-volume=18.2.4-820-g48a7de7e-1jammy radosgw=18.2.4-820-g48a7de7e-1jammy python3-rados=18.2.4-820-g48a7de7e-1jammy python3-rgw=18.2.4-820-g48a7de7e-1jammy python3-cephfs=18.2.4-820-g48a7de7e-1jammy python3-rbd=18.2.4-820-g48a7de7e-1jammy libcephfs2=18.2.4-820-g48a7de7e-1jammy libcephfs-dev=18.2.4-820-g48a7de7e-1jammy librados2=18.2.4-820-g48a7de7e-1jammy librbd1=18.2.4-820-g48a7de7e-1jammy rbd-fuse=18.2.4-820-g48a7de7e-1jammy rbd-nbd=18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:15:45.208 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-13T05:15:45.212 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-09-13T05:15:45.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-820-g48a7de7e-1jammy cephadm=18.2.4-820-g48a7de7e-1jammy ceph-mds=18.2.4-820-g48a7de7e-1jammy ceph-mgr=18.2.4-820-g48a7de7e-1jammy ceph-common=18.2.4-820-g48a7de7e-1jammy ceph-fuse=18.2.4-820-g48a7de7e-1jammy ceph-test=18.2.4-820-g48a7de7e-1jammy ceph-volume=18.2.4-820-g48a7de7e-1jammy radosgw=18.2.4-820-g48a7de7e-1jammy python3-rados=18.2.4-820-g48a7de7e-1jammy python3-rgw=18.2.4-820-g48a7de7e-1jammy python3-cephfs=18.2.4-820-g48a7de7e-1jammy python3-rbd=18.2.4-820-g48a7de7e-1jammy libcephfs2=18.2.4-820-g48a7de7e-1jammy libcephfs-dev=18.2.4-820-g48a7de7e-1jammy librados2=18.2.4-820-g48a7de7e-1jammy librbd1=18.2.4-820-g48a7de7e-1jammy rbd-fuse=18.2.4-820-g48a7de7e-1jammy rbd-nbd=18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:15:45.227 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-09-13T05:15:45.229 DEBUG:teuthology.orchestra.run.smithi079:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-820-g48a7de7e-1jammy cephadm=18.2.4-820-g48a7de7e-1jammy ceph-mds=18.2.4-820-g48a7de7e-1jammy ceph-mgr=18.2.4-820-g48a7de7e-1jammy ceph-common=18.2.4-820-g48a7de7e-1jammy ceph-fuse=18.2.4-820-g48a7de7e-1jammy ceph-test=18.2.4-820-g48a7de7e-1jammy ceph-volume=18.2.4-820-g48a7de7e-1jammy radosgw=18.2.4-820-g48a7de7e-1jammy python3-rados=18.2.4-820-g48a7de7e-1jammy python3-rgw=18.2.4-820-g48a7de7e-1jammy python3-cephfs=18.2.4-820-g48a7de7e-1jammy python3-rbd=18.2.4-820-g48a7de7e-1jammy libcephfs2=18.2.4-820-g48a7de7e-1jammy libcephfs-dev=18.2.4-820-g48a7de7e-1jammy librados2=18.2.4-820-g48a7de7e-1jammy librbd1=18.2.4-820-g48a7de7e-1jammy rbd-fuse=18.2.4-820-g48a7de7e-1jammy rbd-nbd=18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:15:45.292 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-13T05:15:45.293 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-09-13T05:15:45.412 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2024-09-13T05:15:45.413 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2024-09-13T05:15:45.504 INFO:teuthology.orchestra.run.smithi079.stdout:Building dependency tree... 2024-09-13T05:15:45.505 INFO:teuthology.orchestra.run.smithi079.stdout:Reading state information... 2024-09-13T05:15:45.507 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2024-09-13T05:15:45.508 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2024-09-13T05:15:45.597 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T05:15:45.597 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T05:15:45.597 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T05:15:45.597 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T05:15:45.598 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T05:15:45.599 INFO:teuthology.orchestra.run.smithi122.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T05:15:45.599 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T05:15:45.599 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2024-09-13T05:15:45.599 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T05:15:45.599 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T05:15:45.599 INFO:teuthology.orchestra.run.smithi122.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-09-13T05:15:45.599 INFO:teuthology.orchestra.run.smithi122.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T05:15:45.600 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-09-13T05:15:45.600 INFO:teuthology.orchestra.run.smithi122.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T05:15:45.600 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T05:15:45.600 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T05:15:45.600 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T05:15:45.600 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T05:15:45.600 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T05:15:45.600 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T05:15:45.601 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-09-13T05:15:45.602 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2024-09-13T05:15:45.602 INFO:teuthology.orchestra.run.smithi122.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-09-13T05:15:45.602 INFO:teuthology.orchestra.run.smithi122.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-09-13T05:15:45.602 INFO:teuthology.orchestra.run.smithi122.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-09-13T05:15:45.602 INFO:teuthology.orchestra.run.smithi122.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-09-13T05:15:45.602 INFO:teuthology.orchestra.run.smithi122.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-09-13T05:15:45.602 INFO:teuthology.orchestra.run.smithi122.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-09-13T05:15:45.602 INFO:teuthology.orchestra.run.smithi122.stdout: python3-watchdog 2024-09-13T05:15:45.603 INFO:teuthology.orchestra.run.smithi122.stdout:Recommended packages: 2024-09-13T05:15:45.603 INFO:teuthology.orchestra.run.smithi122.stdout: btrfs-tools 2024-09-13T05:15:45.701 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T05:15:45.702 INFO:teuthology.orchestra.run.smithi079.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T05:15:45.702 INFO:teuthology.orchestra.run.smithi079.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T05:15:45.702 INFO:teuthology.orchestra.run.smithi079.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T05:15:45.703 INFO:teuthology.orchestra.run.smithi079.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T05:15:45.703 INFO:teuthology.orchestra.run.smithi079.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T05:15:45.703 INFO:teuthology.orchestra.run.smithi079.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T05:15:45.703 INFO:teuthology.orchestra.run.smithi079.stdout:The following additional packages will be installed: 2024-09-13T05:15:45.703 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T05:15:45.705 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T05:15:45.705 INFO:teuthology.orchestra.run.smithi079.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-09-13T05:15:45.705 INFO:teuthology.orchestra.run.smithi079.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T05:15:45.705 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-09-13T05:15:45.705 INFO:teuthology.orchestra.run.smithi079.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T05:15:45.705 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T05:15:45.705 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T05:15:45.705 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T05:15:45.706 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-09-13T05:15:45.707 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-13T05:15:45.708 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd socat xmlstarlet 2024-09-13T05:15:45.709 INFO:teuthology.orchestra.run.smithi079.stdout:Suggested packages: 2024-09-13T05:15:45.709 INFO:teuthology.orchestra.run.smithi079.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-09-13T05:15:45.709 INFO:teuthology.orchestra.run.smithi079.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-09-13T05:15:45.709 INFO:teuthology.orchestra.run.smithi079.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-09-13T05:15:45.709 INFO:teuthology.orchestra.run.smithi079.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-09-13T05:15:45.709 INFO:teuthology.orchestra.run.smithi079.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-09-13T05:15:45.709 INFO:teuthology.orchestra.run.smithi079.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-09-13T05:15:45.709 INFO:teuthology.orchestra.run.smithi079.stdout: python3-watchdog 2024-09-13T05:15:45.710 INFO:teuthology.orchestra.run.smithi079.stdout:Recommended packages: 2024-09-13T05:15:45.710 INFO:teuthology.orchestra.run.smithi079.stdout: btrfs-tools 2024-09-13T05:15:45.726 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T05:15:45.726 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T05:15:45.726 INFO:teuthology.orchestra.run.smithi016.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T05:15:45.726 INFO:teuthology.orchestra.run.smithi016.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T05:15:45.727 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T05:15:45.727 INFO:teuthology.orchestra.run.smithi016.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T05:15:45.727 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T05:15:45.728 INFO:teuthology.orchestra.run.smithi016.stdout:The following additional packages will be installed: 2024-09-13T05:15:45.728 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T05:15:45.728 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T05:15:45.728 INFO:teuthology.orchestra.run.smithi016.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-09-13T05:15:45.728 INFO:teuthology.orchestra.run.smithi016.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T05:15:45.729 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-09-13T05:15:45.729 INFO:teuthology.orchestra.run.smithi016.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T05:15:45.729 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T05:15:45.729 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T05:15:45.729 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T05:15:45.730 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout:Suggested packages: 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout: python3-watchdog 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout:Recommended packages: 2024-09-13T05:15:45.731 INFO:teuthology.orchestra.run.smithi016.stdout: btrfs-tools 2024-09-13T05:15:45.765 INFO:teuthology.orchestra.run.smithi122.stdout:0 upgraded, 110 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T05:15:45.765 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 163 MB of archives. 2024-09-13T05:15:45.765 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 746 MB of additional disk space will be used. 2024-09-13T05:15:45.765 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-09-13T05:15:45.809 INFO:teuthology.orchestra.run.smithi079.stdout:The following NEW packages will be installed: 2024-09-13T05:15:45.809 INFO:teuthology.orchestra.run.smithi079.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T05:15:45.809 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T05:15:45.810 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T05:15:45.810 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-09-13T05:15:45.810 INFO:teuthology.orchestra.run.smithi079.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-13T05:15:45.810 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-09-13T05:15:45.810 INFO:teuthology.orchestra.run.smithi079.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-13T05:15:45.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-13T05:15:45.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-13T05:15:45.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T05:15:45.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T05:15:45.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T05:15:45.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T05:15:45.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-13T05:15:45.812 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd socat xmlstarlet 2024-09-13T05:15:45.822 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T05:15:45.842 INFO:teuthology.orchestra.run.smithi016.stdout:The following NEW packages will be installed: 2024-09-13T05:15:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T05:15:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T05:15:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T05:15:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-09-13T05:15:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-13T05:15:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-13T05:15:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-13T05:15:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T05:15:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T05:15:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T05:15:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T05:15:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-13T05:15:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd socat xmlstarlet 2024-09-13T05:15:45.856 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.4-820-g48a7de7e-1jammy [3,451 kB] 2024-09-13T05:15:45.859 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T05:15:45.862 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T05:15:45.870 INFO:teuthology.orchestra.run.smithi079.stdout:0 upgraded, 110 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T05:15:45.870 INFO:teuthology.orchestra.run.smithi079.stdout:Need to get 163 MB of archives. 2024-09-13T05:15:45.870 INFO:teuthology.orchestra.run.smithi079.stdout:After this operation, 746 MB of additional disk space will be used. 2024-09-13T05:15:45.870 INFO:teuthology.orchestra.run.smithi079.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-09-13T05:15:45.877 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T05:15:45.904 INFO:teuthology.orchestra.run.smithi016.stdout:0 upgraded, 110 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T05:15:45.905 INFO:teuthology.orchestra.run.smithi016.stdout:Need to get 163 MB of archives. 2024-09-13T05:15:45.905 INFO:teuthology.orchestra.run.smithi016.stdout:After this operation, 746 MB of additional disk space will be used. 2024-09-13T05:15:45.905 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-09-13T05:15:45.927 INFO:teuthology.orchestra.run.smithi079.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T05:15:45.929 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T05:15:45.933 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T05:15:45.945 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T05:15:45.949 INFO:teuthology.orchestra.run.smithi122.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T05:15:45.949 INFO:teuthology.orchestra.run.smithi122.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T05:15:45.950 INFO:teuthology.orchestra.run.smithi122.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T05:15:45.957 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.4-820-g48a7de7e-1jammy [3,451 kB] 2024-09-13T05:15:45.959 INFO:teuthology.orchestra.run.smithi122.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T05:15:45.959 INFO:teuthology.orchestra.run.smithi122.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T05:15:45.959 INFO:teuthology.orchestra.run.smithi122.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T05:15:45.961 INFO:teuthology.orchestra.run.smithi122.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T05:15:45.962 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T05:15:45.964 INFO:teuthology.orchestra.run.smithi079.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T05:15:45.967 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T05:15:45.968 INFO:teuthology.orchestra.run.smithi122.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T05:15:45.968 INFO:teuthology.orchestra.run.smithi122.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T05:15:45.969 INFO:teuthology.orchestra.run.smithi122.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T05:15:45.982 INFO:teuthology.orchestra.run.smithi079.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T05:15:45.985 INFO:teuthology.orchestra.run.smithi122.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T05:15:45.985 INFO:teuthology.orchestra.run.smithi122.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T05:15:45.985 INFO:teuthology.orchestra.run.smithi122.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T05:15:45.988 INFO:teuthology.orchestra.run.smithi122.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T05:15:45.989 INFO:teuthology.orchestra.run.smithi122.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T05:15:45.989 INFO:teuthology.orchestra.run.smithi122.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T05:15:45.989 INFO:teuthology.orchestra.run.smithi122.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T05:15:45.991 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.4-820-g48a7de7e-1jammy [3,451 kB] 2024-09-13T05:15:45.999 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T05:15:46.002 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T05:15:46.002 INFO:teuthology.orchestra.run.smithi122.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T05:15:46.002 INFO:teuthology.orchestra.run.smithi122.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T05:15:46.003 INFO:teuthology.orchestra.run.smithi122.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T05:15:46.017 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T05:15:46.019 INFO:teuthology.orchestra.run.smithi122.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T05:15:46.020 INFO:teuthology.orchestra.run.smithi122.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T05:15:46.020 INFO:teuthology.orchestra.run.smithi122.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T05:15:46.026 INFO:teuthology.orchestra.run.smithi122.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T05:15:46.026 INFO:teuthology.orchestra.run.smithi122.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T05:15:46.026 INFO:teuthology.orchestra.run.smithi122.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T05:15:46.027 INFO:teuthology.orchestra.run.smithi122.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T05:15:46.033 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T05:15:46.036 INFO:teuthology.orchestra.run.smithi122.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T05:15:46.037 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T05:15:46.038 INFO:teuthology.orchestra.run.smithi122.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T05:15:46.039 INFO:teuthology.orchestra.run.smithi122.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T05:15:46.040 INFO:teuthology.orchestra.run.smithi122.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.4-820-g48a7de7e-1jammy [3,108 kB] 2024-09-13T05:15:46.049 INFO:teuthology.orchestra.run.smithi079.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T05:15:46.053 INFO:teuthology.orchestra.run.smithi122.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T05:15:46.053 INFO:teuthology.orchestra.run.smithi079.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T05:15:46.054 INFO:teuthology.orchestra.run.smithi079.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T05:15:46.054 INFO:teuthology.orchestra.run.smithi122.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T05:15:46.054 INFO:teuthology.orchestra.run.smithi079.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T05:15:46.056 INFO:teuthology.orchestra.run.smithi122.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T05:15:46.057 INFO:teuthology.orchestra.run.smithi122.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T05:15:46.058 INFO:teuthology.orchestra.run.smithi122.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T05:15:46.060 INFO:teuthology.orchestra.run.smithi122.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T05:15:46.062 INFO:teuthology.orchestra.run.smithi079.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T05:15:46.063 INFO:teuthology.orchestra.run.smithi079.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T05:15:46.063 INFO:teuthology.orchestra.run.smithi079.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T05:15:46.065 INFO:teuthology.orchestra.run.smithi079.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T05:15:46.069 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T05:15:46.073 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T05:15:46.076 INFO:teuthology.orchestra.run.smithi079.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T05:15:46.077 INFO:teuthology.orchestra.run.smithi079.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T05:15:46.077 INFO:teuthology.orchestra.run.smithi079.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T05:15:46.086 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T05:15:46.090 INFO:teuthology.orchestra.run.smithi016.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T05:15:46.090 INFO:teuthology.orchestra.run.smithi016.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T05:15:46.091 INFO:teuthology.orchestra.run.smithi016.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T05:15:46.093 INFO:teuthology.orchestra.run.smithi079.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T05:15:46.094 INFO:teuthology.orchestra.run.smithi079.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T05:15:46.094 INFO:teuthology.orchestra.run.smithi079.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T05:15:46.097 INFO:teuthology.orchestra.run.smithi079.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T05:15:46.098 INFO:teuthology.orchestra.run.smithi079.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T05:15:46.098 INFO:teuthology.orchestra.run.smithi079.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T05:15:46.098 INFO:teuthology.orchestra.run.smithi079.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T05:15:46.100 INFO:teuthology.orchestra.run.smithi016.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T05:15:46.100 INFO:teuthology.orchestra.run.smithi016.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T05:15:46.101 INFO:teuthology.orchestra.run.smithi016.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T05:15:46.102 INFO:teuthology.orchestra.run.smithi016.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T05:15:46.105 INFO:teuthology.orchestra.run.smithi122.stdout:Get:47 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.4-820-g48a7de7e-1jammy [683 kB] 2024-09-13T05:15:46.107 INFO:teuthology.orchestra.run.smithi016.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T05:15:46.107 INFO:teuthology.orchestra.run.smithi016.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T05:15:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T05:15:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T05:15:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T05:15:46.114 INFO:teuthology.orchestra.run.smithi122.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T05:15:46.115 INFO:teuthology.orchestra.run.smithi122.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.4-820-g48a7de7e-1jammy [357 kB] 2024-09-13T05:15:46.115 INFO:teuthology.orchestra.run.smithi122.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T05:15:46.116 INFO:teuthology.orchestra.run.smithi122.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T05:15:46.117 INFO:teuthology.orchestra.run.smithi122.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T05:15:46.117 INFO:teuthology.orchestra.run.smithi122.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T05:15:46.124 INFO:teuthology.orchestra.run.smithi016.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T05:15:46.125 INFO:teuthology.orchestra.run.smithi016.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T05:15:46.125 INFO:teuthology.orchestra.run.smithi016.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T05:15:46.125 INFO:teuthology.orchestra.run.smithi016.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T05:15:46.127 INFO:teuthology.orchestra.run.smithi079.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T05:15:46.128 INFO:teuthology.orchestra.run.smithi122.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T05:15:46.128 INFO:teuthology.orchestra.run.smithi122.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.4-820-g48a7de7e-1jammy [32.8 kB] 2024-09-13T05:15:46.128 INFO:teuthology.orchestra.run.smithi122.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T05:15:46.129 INFO:teuthology.orchestra.run.smithi122.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.4-820-g48a7de7e-1jammy [180 kB] 2024-09-13T05:15:46.130 INFO:teuthology.orchestra.run.smithi079.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T05:15:46.130 INFO:teuthology.orchestra.run.smithi079.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T05:15:46.130 INFO:teuthology.orchestra.run.smithi016.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T05:15:46.130 INFO:teuthology.orchestra.run.smithi016.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T05:15:46.130 INFO:teuthology.orchestra.run.smithi016.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T05:15:46.130 INFO:teuthology.orchestra.run.smithi016.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T05:15:46.131 INFO:teuthology.orchestra.run.smithi122.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T05:15:46.131 INFO:teuthology.orchestra.run.smithi122.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T05:15:46.131 INFO:teuthology.orchestra.run.smithi122.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.4-820-g48a7de7e-1jammy [63.8 kB] 2024-09-13T05:15:46.132 INFO:teuthology.orchestra.run.smithi122.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.4-820-g48a7de7e-1jammy [334 kB] 2024-09-13T05:15:46.133 INFO:teuthology.orchestra.run.smithi122.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T05:15:46.134 INFO:teuthology.orchestra.run.smithi079.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T05:15:46.134 INFO:teuthology.orchestra.run.smithi079.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T05:15:46.135 INFO:teuthology.orchestra.run.smithi079.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T05:15:46.135 INFO:teuthology.orchestra.run.smithi079.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T05:15:46.141 INFO:teuthology.orchestra.run.smithi016.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T05:15:46.141 INFO:teuthology.orchestra.run.smithi016.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T05:15:46.144 INFO:teuthology.orchestra.run.smithi079.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T05:15:46.146 INFO:teuthology.orchestra.run.smithi079.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T05:15:46.147 INFO:teuthology.orchestra.run.smithi079.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T05:15:46.148 INFO:teuthology.orchestra.run.smithi122.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.4-820-g48a7de7e-1jammy [5,934 kB] 2024-09-13T05:15:46.152 INFO:teuthology.orchestra.run.smithi122.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T05:15:46.152 INFO:teuthology.orchestra.run.smithi122.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T05:15:46.152 INFO:teuthology.orchestra.run.smithi122.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T05:15:46.158 INFO:teuthology.orchestra.run.smithi016.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T05:15:46.160 INFO:teuthology.orchestra.run.smithi016.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T05:15:46.160 INFO:teuthology.orchestra.run.smithi016.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T05:15:46.161 INFO:teuthology.orchestra.run.smithi016.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T05:15:46.161 INFO:teuthology.orchestra.run.smithi079.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T05:15:46.162 INFO:teuthology.orchestra.run.smithi079.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T05:15:46.165 INFO:teuthology.orchestra.run.smithi079.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T05:15:46.165 INFO:teuthology.orchestra.run.smithi079.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T05:15:46.166 INFO:teuthology.orchestra.run.smithi079.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T05:15:46.166 INFO:teuthology.orchestra.run.smithi016.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T05:15:46.167 INFO:teuthology.orchestra.run.smithi016.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T05:15:46.167 INFO:teuthology.orchestra.run.smithi016.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T05:15:46.168 INFO:teuthology.orchestra.run.smithi016.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T05:15:46.168 INFO:teuthology.orchestra.run.smithi122.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T05:15:46.169 INFO:teuthology.orchestra.run.smithi122.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T05:15:46.169 INFO:teuthology.orchestra.run.smithi079.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T05:15:46.170 INFO:teuthology.orchestra.run.smithi122.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T05:15:46.170 INFO:teuthology.orchestra.run.smithi122.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T05:15:46.170 INFO:teuthology.orchestra.run.smithi122.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T05:15:46.171 INFO:teuthology.orchestra.run.smithi122.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T05:15:46.171 INFO:teuthology.orchestra.run.smithi122.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T05:15:46.175 INFO:teuthology.orchestra.run.smithi016.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T05:15:46.177 INFO:teuthology.orchestra.run.smithi016.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T05:15:46.186 INFO:teuthology.orchestra.run.smithi122.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T05:15:46.187 INFO:teuthology.orchestra.run.smithi016.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T05:15:46.188 INFO:teuthology.orchestra.run.smithi016.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T05:15:46.189 INFO:teuthology.orchestra.run.smithi016.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T05:15:46.189 INFO:teuthology.orchestra.run.smithi122.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T05:15:46.192 INFO:teuthology.orchestra.run.smithi016.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T05:15:46.192 INFO:teuthology.orchestra.run.smithi016.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T05:15:46.193 INFO:teuthology.orchestra.run.smithi016.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T05:15:46.197 INFO:teuthology.orchestra.run.smithi016.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T05:15:46.203 INFO:teuthology.orchestra.run.smithi122.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T05:15:46.205 INFO:teuthology.orchestra.run.smithi122.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T05:15:46.205 INFO:teuthology.orchestra.run.smithi122.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T05:15:46.208 INFO:teuthology.orchestra.run.smithi122.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T05:15:46.211 INFO:teuthology.orchestra.run.smithi122.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T05:15:46.220 INFO:teuthology.orchestra.run.smithi079.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T05:15:46.221 INFO:teuthology.orchestra.run.smithi079.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T05:15:46.222 INFO:teuthology.orchestra.run.smithi079.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T05:15:46.222 INFO:teuthology.orchestra.run.smithi079.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T05:15:46.223 INFO:teuthology.orchestra.run.smithi079.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T05:15:46.223 INFO:teuthology.orchestra.run.smithi079.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T05:15:46.224 INFO:teuthology.orchestra.run.smithi079.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T05:15:46.226 INFO:teuthology.orchestra.run.smithi079.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T05:15:46.226 INFO:teuthology.orchestra.run.smithi079.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T05:15:46.240 INFO:teuthology.orchestra.run.smithi079.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T05:15:46.250 INFO:teuthology.orchestra.run.smithi122.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T05:15:46.251 INFO:teuthology.orchestra.run.smithi016.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T05:15:46.252 INFO:teuthology.orchestra.run.smithi016.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T05:15:46.253 INFO:teuthology.orchestra.run.smithi122.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T05:15:46.254 INFO:teuthology.orchestra.run.smithi016.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T05:15:46.254 INFO:teuthology.orchestra.run.smithi016.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T05:15:46.254 INFO:teuthology.orchestra.run.smithi122.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T05:15:46.255 INFO:teuthology.orchestra.run.smithi016.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T05:15:46.255 INFO:teuthology.orchestra.run.smithi016.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T05:15:46.256 INFO:teuthology.orchestra.run.smithi122.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T05:15:46.256 INFO:teuthology.orchestra.run.smithi016.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T05:15:46.258 INFO:teuthology.orchestra.run.smithi016.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T05:15:46.258 INFO:teuthology.orchestra.run.smithi016.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T05:15:46.262 INFO:teuthology.orchestra.run.smithi122.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T05:15:46.262 INFO:teuthology.orchestra.run.smithi122.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T05:15:46.263 INFO:teuthology.orchestra.run.smithi122.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T05:15:46.263 INFO:teuthology.orchestra.run.smithi079.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T05:15:46.265 INFO:teuthology.orchestra.run.smithi079.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T05:15:46.265 INFO:teuthology.orchestra.run.smithi079.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T05:15:46.267 INFO:teuthology.orchestra.run.smithi079.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.4-820-g48a7de7e-1jammy [3,108 kB] 2024-09-13T05:15:46.271 INFO:teuthology.orchestra.run.smithi016.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T05:15:46.281 INFO:teuthology.orchestra.run.smithi079.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T05:15:46.281 INFO:teuthology.orchestra.run.smithi079.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T05:15:46.281 INFO:teuthology.orchestra.run.smithi079.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T05:15:46.282 INFO:teuthology.orchestra.run.smithi079.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T05:15:46.282 INFO:teuthology.orchestra.run.smithi079.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T05:15:46.283 INFO:teuthology.orchestra.run.smithi079.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T05:15:46.283 INFO:teuthology.orchestra.run.smithi079.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T05:15:46.288 INFO:teuthology.orchestra.run.smithi016.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T05:15:46.290 INFO:teuthology.orchestra.run.smithi016.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T05:15:46.290 INFO:teuthology.orchestra.run.smithi016.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T05:15:46.298 INFO:teuthology.orchestra.run.smithi079.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T05:15:46.302 INFO:teuthology.orchestra.run.smithi079.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T05:15:46.306 INFO:teuthology.orchestra.run.smithi016.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T05:15:46.306 INFO:teuthology.orchestra.run.smithi016.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T05:15:46.307 INFO:teuthology.orchestra.run.smithi016.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T05:15:46.307 INFO:teuthology.orchestra.run.smithi016.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T05:15:46.308 INFO:teuthology.orchestra.run.smithi016.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T05:15:46.308 INFO:teuthology.orchestra.run.smithi016.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T05:15:46.309 INFO:teuthology.orchestra.run.smithi016.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T05:15:46.315 INFO:teuthology.orchestra.run.smithi079.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T05:15:46.317 INFO:teuthology.orchestra.run.smithi079.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T05:15:46.317 INFO:teuthology.orchestra.run.smithi079.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T05:15:46.320 INFO:teuthology.orchestra.run.smithi079.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T05:15:46.322 INFO:teuthology.orchestra.run.smithi079.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T05:15:46.324 INFO:teuthology.orchestra.run.smithi016.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T05:15:46.327 INFO:teuthology.orchestra.run.smithi079.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T05:15:46.328 INFO:teuthology.orchestra.run.smithi016.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T05:15:46.329 INFO:teuthology.orchestra.run.smithi079.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T05:15:46.330 INFO:teuthology.orchestra.run.smithi079.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T05:15:46.330 INFO:teuthology.orchestra.run.smithi016.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T05:15:46.332 INFO:teuthology.orchestra.run.smithi079.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T05:15:46.339 INFO:teuthology.orchestra.run.smithi079.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T05:15:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T05:15:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T05:15:46.343 INFO:teuthology.orchestra.run.smithi122.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.4-820-g48a7de7e-1jammy [112 kB] 2024-09-13T05:15:46.344 INFO:teuthology.orchestra.run.smithi016.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T05:15:46.345 INFO:teuthology.orchestra.run.smithi122.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.4-820-g48a7de7e-1jammy [445 kB] 2024-09-13T05:15:46.347 INFO:teuthology.orchestra.run.smithi016.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T05:15:46.349 INFO:teuthology.orchestra.run.smithi079.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T05:15:46.350 INFO:teuthology.orchestra.run.smithi079.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T05:15:46.352 INFO:teuthology.orchestra.run.smithi016.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T05:15:46.352 INFO:teuthology.orchestra.run.smithi122.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.4-820-g48a7de7e-1jammy [22.0 MB] 2024-09-13T05:15:46.353 INFO:teuthology.orchestra.run.smithi016.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T05:15:46.354 INFO:teuthology.orchestra.run.smithi016.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T05:15:46.358 INFO:teuthology.orchestra.run.smithi016.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T05:15:46.364 INFO:teuthology.orchestra.run.smithi016.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T05:15:46.364 INFO:teuthology.orchestra.run.smithi016.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T05:15:46.374 INFO:teuthology.orchestra.run.smithi016.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T05:15:46.397 INFO:teuthology.orchestra.run.smithi079.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.4-820-g48a7de7e-1jammy [683 kB] 2024-09-13T05:15:46.446 INFO:teuthology.orchestra.run.smithi079.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.4-820-g48a7de7e-1jammy [357 kB] 2024-09-13T05:15:46.454 INFO:teuthology.orchestra.run.smithi079.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.4-820-g48a7de7e-1jammy [32.8 kB] 2024-09-13T05:15:46.455 INFO:teuthology.orchestra.run.smithi079.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.4-820-g48a7de7e-1jammy [180 kB] 2024-09-13T05:15:46.473 INFO:teuthology.orchestra.run.smithi079.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.4-820-g48a7de7e-1jammy [63.8 kB] 2024-09-13T05:15:46.475 INFO:teuthology.orchestra.run.smithi079.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.4-820-g48a7de7e-1jammy [334 kB] 2024-09-13T05:15:46.501 INFO:teuthology.orchestra.run.smithi079.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.4-820-g48a7de7e-1jammy [5,934 kB] 2024-09-13T05:15:46.855 INFO:teuthology.orchestra.run.smithi079.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.4-820-g48a7de7e-1jammy [112 kB] 2024-09-13T05:15:46.858 INFO:teuthology.orchestra.run.smithi079.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.4-820-g48a7de7e-1jammy [445 kB] 2024-09-13T05:15:46.865 INFO:teuthology.orchestra.run.smithi079.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.4-820-g48a7de7e-1jammy [22.0 MB] 2024-09-13T05:15:46.943 INFO:teuthology.orchestra.run.smithi122.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.4-820-g48a7de7e-1jammy [5,005 kB] 2024-09-13T05:15:47.006 INFO:teuthology.orchestra.run.smithi016.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.4-820-g48a7de7e-1jammy [3,108 kB] 2024-09-13T05:15:47.029 INFO:teuthology.orchestra.run.smithi122.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.4-820-g48a7de7e-1jammy [241 kB] 2024-09-13T05:15:47.033 INFO:teuthology.orchestra.run.smithi122.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.4-820-g48a7de7e-1jammy [130 kB] 2024-09-13T05:15:47.036 INFO:teuthology.orchestra.run.smithi122.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.4-820-g48a7de7e-1jammy [1,494 kB] 2024-09-13T05:15:47.081 INFO:teuthology.orchestra.run.smithi122.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.4-820-g48a7de7e-1jammy [6,085 kB] 2024-09-13T05:15:47.271 INFO:teuthology.orchestra.run.smithi122.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.4-820-g48a7de7e-1jammy [22.4 MB] 2024-09-13T05:15:47.570 INFO:teuthology.orchestra.run.smithi016.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.4-820-g48a7de7e-1jammy [683 kB] 2024-09-13T05:15:47.680 INFO:teuthology.orchestra.run.smithi016.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.4-820-g48a7de7e-1jammy [357 kB] 2024-09-13T05:15:47.736 INFO:teuthology.orchestra.run.smithi016.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.4-820-g48a7de7e-1jammy [32.8 kB] 2024-09-13T05:15:47.737 INFO:teuthology.orchestra.run.smithi016.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.4-820-g48a7de7e-1jammy [180 kB] 2024-09-13T05:15:47.788 INFO:teuthology.orchestra.run.smithi016.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.4-820-g48a7de7e-1jammy [63.8 kB] 2024-09-13T05:15:47.790 INFO:teuthology.orchestra.run.smithi016.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.4-820-g48a7de7e-1jammy [334 kB] 2024-09-13T05:15:47.846 INFO:teuthology.orchestra.run.smithi016.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.4-820-g48a7de7e-1jammy [5,934 kB] 2024-09-13T05:15:47.940 INFO:teuthology.orchestra.run.smithi079.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.4-820-g48a7de7e-1jammy [5,005 kB] 2024-09-13T05:15:47.996 INFO:teuthology.orchestra.run.smithi122.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.4-820-g48a7de7e-1jammy [14.2 kB] 2024-09-13T05:15:47.996 INFO:teuthology.orchestra.run.smithi122.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.4-820-g48a7de7e-1jammy [834 kB] 2024-09-13T05:15:48.011 INFO:teuthology.orchestra.run.smithi122.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.4-820-g48a7de7e-1jammy [2,159 kB] 2024-09-13T05:15:48.046 INFO:teuthology.orchestra.run.smithi122.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.4-820-g48a7de7e-1jammy [240 kB] 2024-09-13T05:15:48.050 INFO:teuthology.orchestra.run.smithi122.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.4-820-g48a7de7e-1jammy [145 kB] 2024-09-13T05:15:48.052 INFO:teuthology.orchestra.run.smithi122.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.4-820-g48a7de7e-1jammy [2,233 kB] 2024-09-13T05:15:48.060 INFO:teuthology.orchestra.run.smithi079.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.4-820-g48a7de7e-1jammy [241 kB] 2024-09-13T05:15:48.077 INFO:teuthology.orchestra.run.smithi079.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.4-820-g48a7de7e-1jammy [130 kB] 2024-09-13T05:15:48.090 INFO:teuthology.orchestra.run.smithi122.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.4-820-g48a7de7e-1jammy [8,625 kB] 2024-09-13T05:15:48.099 INFO:teuthology.orchestra.run.smithi079.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.4-820-g48a7de7e-1jammy [1,494 kB] 2024-09-13T05:15:48.162 INFO:teuthology.orchestra.run.smithi079.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.4-820-g48a7de7e-1jammy [6,085 kB] 2024-09-13T05:15:48.352 INFO:teuthology.orchestra.run.smithi122.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.4-820-g48a7de7e-1jammy [14.3 kB] 2024-09-13T05:15:48.353 INFO:teuthology.orchestra.run.smithi122.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.4-820-g48a7de7e-1jammy [42.8 MB] 2024-09-13T05:15:48.458 INFO:teuthology.orchestra.run.smithi079.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.4-820-g48a7de7e-1jammy [22.4 MB] 2024-09-13T05:15:48.870 INFO:teuthology.orchestra.run.smithi016.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.4-820-g48a7de7e-1jammy [112 kB] 2024-09-13T05:15:48.892 INFO:teuthology.orchestra.run.smithi016.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.4-820-g48a7de7e-1jammy [445 kB] 2024-09-13T05:15:48.957 INFO:teuthology.orchestra.run.smithi016.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.4-820-g48a7de7e-1jammy [22.0 MB] 2024-09-13T05:15:49.648 INFO:teuthology.orchestra.run.smithi079.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.4-820-g48a7de7e-1jammy [14.2 kB] 2024-09-13T05:15:49.649 INFO:teuthology.orchestra.run.smithi079.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.4-820-g48a7de7e-1jammy [834 kB] 2024-09-13T05:15:49.665 INFO:teuthology.orchestra.run.smithi079.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.4-820-g48a7de7e-1jammy [2,159 kB] 2024-09-13T05:15:49.704 INFO:teuthology.orchestra.run.smithi079.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.4-820-g48a7de7e-1jammy [240 kB] 2024-09-13T05:15:49.707 INFO:teuthology.orchestra.run.smithi079.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.4-820-g48a7de7e-1jammy [145 kB] 2024-09-13T05:15:49.709 INFO:teuthology.orchestra.run.smithi079.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.4-820-g48a7de7e-1jammy [2,233 kB] 2024-09-13T05:15:49.740 INFO:teuthology.orchestra.run.smithi122.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.4-820-g48a7de7e-1jammy [122 kB] 2024-09-13T05:15:49.743 INFO:teuthology.orchestra.run.smithi122.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.4-820-g48a7de7e-1jammy [38.3 kB] 2024-09-13T05:15:49.744 INFO:teuthology.orchestra.run.smithi122.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.4-820-g48a7de7e-1jammy [11.7 MB] 2024-09-13T05:15:49.744 INFO:teuthology.orchestra.run.smithi079.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.4-820-g48a7de7e-1jammy [8,625 kB] 2024-09-13T05:15:50.012 INFO:teuthology.orchestra.run.smithi122.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.4-820-g48a7de7e-1jammy [93.1 kB] 2024-09-13T05:15:50.014 INFO:teuthology.orchestra.run.smithi122.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 18.2.4-820-g48a7de7e-1jammy [180 kB] 2024-09-13T05:15:50.060 INFO:teuthology.orchestra.run.smithi079.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.4-820-g48a7de7e-1jammy [14.3 kB] 2024-09-13T05:15:50.061 INFO:teuthology.orchestra.run.smithi079.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.4-820-g48a7de7e-1jammy [42.8 MB] 2024-09-13T05:15:50.544 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 163 MB in 4s (37.8 MB/s) 2024-09-13T05:15:50.800 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libfmt8:amd64. 2024-09-13T05:15:50.901 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T05:15:50.906 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-09-13T05:15:50.990 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-13T05:15:51.234 INFO:teuthology.orchestra.run.smithi016.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.4-820-g48a7de7e-1jammy [5,005 kB] 2024-09-13T05:15:51.293 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T05:15:51.319 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T05:15:51.352 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T05:15:51.485 INFO:teuthology.orchestra.run.smithi016.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.4-820-g48a7de7e-1jammy [241 kB] 2024-09-13T05:15:51.490 INFO:teuthology.orchestra.run.smithi016.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.4-820-g48a7de7e-1jammy [130 kB] 2024-09-13T05:15:51.506 INFO:teuthology.orchestra.run.smithi016.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.4-820-g48a7de7e-1jammy [1,494 kB] 2024-09-13T05:15:51.520 INFO:teuthology.orchestra.run.smithi079.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.4-820-g48a7de7e-1jammy [122 kB] 2024-09-13T05:15:51.524 INFO:teuthology.orchestra.run.smithi079.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.4-820-g48a7de7e-1jammy [38.3 kB] 2024-09-13T05:15:51.525 INFO:teuthology.orchestra.run.smithi079.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.4-820-g48a7de7e-1jammy [11.7 MB] 2024-09-13T05:15:51.589 INFO:teuthology.orchestra.run.smithi016.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.4-820-g48a7de7e-1jammy [6,085 kB] 2024-09-13T05:15:51.672 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T05:15:51.698 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T05:15:51.710 INFO:teuthology.orchestra.run.smithi079.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.4-820-g48a7de7e-1jammy [93.1 kB] 2024-09-13T05:15:51.711 INFO:teuthology.orchestra.run.smithi079.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 18.2.4-820-g48a7de7e-1jammy [180 kB] 2024-09-13T05:15:51.739 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T05:15:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.4-820-g48a7de7e-1jammy [22.4 MB] 2024-09-13T05:15:52.092 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T05:15:52.118 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T05:15:52.159 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T05:15:52.278 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 163 MB in 6s (27.6 MB/s) 2024-09-13T05:15:52.391 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libfmt8:amd64. 2024-09-13T05:15:52.487 INFO:teuthology.orchestra.run.smithi079.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T05:15:52.493 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-09-13T05:15:52.554 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T05:15:52.573 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-13T05:15:52.581 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T05:15:52.621 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:15:52.909 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T05:15:52.936 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T05:15:52.976 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T05:15:52.999 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T05:15:53.026 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T05:15:53.058 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:15:53.277 INFO:teuthology.orchestra.run.smithi016.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.4-820-g48a7de7e-1jammy [14.2 kB] 2024-09-13T05:15:53.278 INFO:teuthology.orchestra.run.smithi016.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.4-820-g48a7de7e-1jammy [834 kB] 2024-09-13T05:15:53.295 INFO:teuthology.orchestra.run.smithi016.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.4-820-g48a7de7e-1jammy [2,159 kB] 2024-09-13T05:15:53.321 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T05:15:53.348 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T05:15:53.388 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T05:15:53.395 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T05:15:53.405 INFO:teuthology.orchestra.run.smithi016.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.4-820-g48a7de7e-1jammy [240 kB] 2024-09-13T05:15:53.411 INFO:teuthology.orchestra.run.smithi016.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.4-820-g48a7de7e-1jammy [145 kB] 2024-09-13T05:15:53.421 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T05:15:53.425 INFO:teuthology.orchestra.run.smithi016.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.4-820-g48a7de7e-1jammy [2,233 kB] 2024-09-13T05:15:53.461 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:15:53.536 INFO:teuthology.orchestra.run.smithi016.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.4-820-g48a7de7e-1jammy [8,625 kB] 2024-09-13T05:15:53.674 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T05:15:53.690 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T05:15:53.724 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T05:15:53.798 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T05:15:53.825 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T05:15:53.865 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T05:15:53.968 INFO:teuthology.orchestra.run.smithi016.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.4-820-g48a7de7e-1jammy [14.3 kB] 2024-09-13T05:15:53.968 INFO:teuthology.orchestra.run.smithi016.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.4-820-g48a7de7e-1jammy [42.8 MB] 2024-09-13T05:15:54.143 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librados2. 2024-09-13T05:15:54.161 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T05:15:54.169 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../008-librados2_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:54.187 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T05:15:54.201 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librados2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:54.236 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:15:54.646 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librbd1. 2024-09-13T05:15:54.647 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T05:15:54.673 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../009-librbd1_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:54.674 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T05:15:54.713 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librbd1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:54.714 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:15:55.025 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T05:15:55.052 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T05:15:55.083 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:15:55.141 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T05:15:55.168 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../010-libcephfs2_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:55.207 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libcephfs2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:55.468 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rados. 2024-09-13T05:15:55.494 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../011-python3-rados_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:55.535 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rados (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:55.720 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T05:15:55.747 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T05:15:55.787 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T05:15:55.855 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T05:15:55.882 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:15:55.938 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-ceph-argparse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:56.106 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librados2. 2024-09-13T05:15:56.132 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../008-librados2_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:56.166 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T05:15:56.173 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librados2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:56.181 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../013-python3-cephfs_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:56.183 INFO:teuthology.orchestra.run.smithi016.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.4-820-g48a7de7e-1jammy [122 kB] 2024-09-13T05:15:56.186 INFO:teuthology.orchestra.run.smithi016.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.4-820-g48a7de7e-1jammy [38.3 kB] 2024-09-13T05:15:56.187 INFO:teuthology.orchestra.run.smithi016.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.4-820-g48a7de7e-1jammy [11.7 MB] 2024-09-13T05:15:56.216 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cephfs (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:56.511 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T05:15:56.537 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:15:56.577 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:56.587 INFO:teuthology.orchestra.run.smithi016.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.4-820-g48a7de7e-1jammy [93.1 kB] 2024-09-13T05:15:56.589 INFO:teuthology.orchestra.run.smithi016.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 18.2.4-820-g48a7de7e-1jammy [180 kB] 2024-09-13T05:15:56.617 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librbd1. 2024-09-13T05:15:56.643 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../009-librbd1_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:56.684 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librbd1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:56.889 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T05:15:56.916 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T05:15:56.956 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T05:15:57.095 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T05:15:57.124 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../010-libcephfs2_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:57.161 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libcephfs2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:57.183 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 163 MB in 11s (15.2 MB/s) 2024-09-13T05:15:57.276 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T05:15:57.302 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T05:15:57.314 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libfmt8:amd64. 2024-09-13T05:15:57.342 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T05:15:57.412 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T05:15:57.417 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-09-13T05:15:57.447 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rados. 2024-09-13T05:15:57.474 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../011-python3-rados_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:57.504 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-13T05:15:57.514 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rados (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:57.570 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T05:15:57.586 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../017-python3-rbd_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:57.620 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:57.824 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T05:15:57.842 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T05:15:57.851 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T05:15:57.868 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:15:57.882 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T05:15:57.908 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-ceph-argparse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:57.965 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T05:15:57.992 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T05:15:58.032 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T05:15:58.144 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T05:15:58.170 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../013-python3-cephfs_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:58.211 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cephfs (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:58.236 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T05:15:58.263 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T05:15:58.302 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T05:15:58.360 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librgw2. 2024-09-13T05:15:58.386 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../019-librgw2_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:58.427 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librgw2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:58.514 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T05:15:58.540 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:15:58.556 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T05:15:58.580 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:58.582 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T05:15:58.614 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T05:15:58.896 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T05:15:58.900 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T05:15:58.911 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:58.926 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T05:15:58.946 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:58.966 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T05:15:59.084 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T05:15:59.111 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T05:15:59.159 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:15:59.282 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T05:15:59.303 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T05:15:59.308 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T05:15:59.318 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T05:15:59.349 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T05:15:59.352 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T05:15:59.529 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T05:15:59.556 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T05:15:59.587 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:15:59.588 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T05:15:59.604 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../017-python3-rbd_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:59.647 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:59.652 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T05:15:59.678 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:15:59.719 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libradosstriper1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:15:59.924 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T05:15:59.951 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T05:15:59.991 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:15:59.999 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T05:16:00.026 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T05:16:00.083 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T05:16:00.362 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T05:16:00.381 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-common. 2024-09-13T05:16:00.389 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T05:16:00.408 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../023-ceph-common_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:00.419 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librgw2. 2024-09-13T05:16:00.435 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T05:16:00.439 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:00.445 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../019-librgw2_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:00.486 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librgw2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:00.739 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librados2. 2024-09-13T05:16:00.766 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../008-librados2_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:00.797 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librados2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:00.971 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T05:16:00.986 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:01.021 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:01.341 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T05:16:01.368 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T05:16:01.407 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T05:16:01.425 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librbd1. 2024-09-13T05:16:01.452 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../009-librbd1_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:01.492 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librbd1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:01.690 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-base. 2024-09-13T05:16:01.710 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T05:16:01.716 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../024-ceph-base_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:01.736 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:01.766 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-base (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:01.777 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libradosstriper1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:01.920 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T05:16:01.947 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../010-libcephfs2_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:01.986 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libcephfs2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:02.205 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-common. 2024-09-13T05:16:02.232 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../023-ceph-common_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:02.272 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rados. 2024-09-13T05:16:02.288 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../011-python3-rados_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:02.322 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rados (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:02.328 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T05:16:02.355 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T05:16:02.394 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T05:16:02.592 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T05:16:02.608 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:02.642 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-ceph-argparse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:02.664 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T05:16:02.691 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T05:16:02.731 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T05:16:02.878 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T05:16:02.905 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../013-python3-cephfs_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:02.944 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cephfs (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:03.059 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T05:16:03.086 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T05:16:03.126 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T05:16:03.264 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T05:16:03.291 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:03.330 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:03.404 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T05:16:03.431 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T05:16:03.470 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T05:16:03.627 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-base. 2024-09-13T05:16:03.644 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T05:16:03.654 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../024-ceph-base_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:03.660 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T05:16:03.694 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T05:16:03.703 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-base (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:03.732 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T05:16:03.758 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T05:16:03.791 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T05:16:04.022 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T05:16:04.049 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T05:16:04.069 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T05:16:04.088 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T05:16:04.095 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T05:16:04.152 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T05:16:04.265 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T05:16:04.280 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T05:16:04.315 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T05:16:04.366 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T05:16:04.393 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../017-python3-rbd_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:04.432 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:04.463 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-portend. 2024-09-13T05:16:04.489 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-09-13T05:16:04.530 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T05:16:04.575 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T05:16:04.602 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T05:16:04.642 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T05:16:04.769 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T05:16:04.796 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T05:16:04.808 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T05:16:04.835 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T05:16:04.835 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T05:16:04.867 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T05:16:04.962 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T05:16:04.988 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T05:16:05.028 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T05:16:05.113 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librgw2. 2024-09-13T05:16:05.140 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../019-librgw2_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:05.145 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T05:16:05.171 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T05:16:05.172 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librgw2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:05.211 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T05:16:05.331 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T05:16:05.358 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T05:16:05.398 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T05:16:05.566 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T05:16:05.592 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T05:16:05.632 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T05:16:05.665 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T05:16:05.682 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:05.724 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:05.867 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T05:16:05.894 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T05:16:05.927 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T05:16:05.933 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T05:16:05.954 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T05:16:05.994 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T05:16:06.052 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T05:16:06.079 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T05:16:06.118 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T05:16:06.255 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T05:16:06.270 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T05:16:06.271 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T05:16:06.296 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T05:16:06.305 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T05:16:06.336 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T05:16:06.588 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T05:16:06.606 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-portend. 2024-09-13T05:16:06.609 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-mako. 2024-09-13T05:16:06.615 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:06.632 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-09-13T05:16:06.635 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T05:16:06.654 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libradosstriper1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:06.664 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T05:16:06.675 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T05:16:06.925 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T05:16:06.951 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T05:16:06.992 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T05:16:06.995 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T05:16:07.022 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T05:16:07.062 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T05:16:07.083 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-common. 2024-09-13T05:16:07.110 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../023-ceph-common_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:07.149 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:07.305 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T05:16:07.332 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T05:16:07.365 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T05:16:07.372 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T05:16:07.392 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T05:16:07.424 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T05:16:07.684 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T05:16:07.702 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-webob. 2024-09-13T05:16:07.713 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T05:16:07.728 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T05:16:07.750 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T05:16:07.768 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T05:16:08.054 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T05:16:08.081 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T05:16:08.113 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T05:16:08.120 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T05:16:08.139 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T05:16:08.183 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T05:16:08.424 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T05:16:08.450 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T05:16:08.455 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-base. 2024-09-13T05:16:08.482 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../024-ceph-base_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:08.490 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T05:16:08.508 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T05:16:08.531 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-base (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:08.535 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T05:16:08.567 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T05:16:08.793 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-mako. 2024-09-13T05:16:08.812 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-paste. 2024-09-13T05:16:08.820 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T05:16:08.838 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T05:16:08.851 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T05:16:08.878 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T05:16:09.155 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T05:16:09.167 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T05:16:09.182 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T05:16:09.182 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T05:16:09.194 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T05:16:09.197 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T05:16:09.222 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T05:16:09.232 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T05:16:09.234 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T05:16:09.477 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T05:16:09.503 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T05:16:09.525 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T05:16:09.537 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T05:16:09.543 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T05:16:09.551 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T05:16:09.564 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T05:16:09.591 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T05:16:09.603 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T05:16:09.846 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T05:16:09.873 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T05:16:09.898 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T05:16:09.911 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-webob. 2024-09-13T05:16:09.913 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T05:16:09.925 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T05:16:09.940 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T05:16:09.965 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T05:16:09.977 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T05:16:10.217 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T05:16:10.233 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T05:16:10.233 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T05:16:10.259 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T05:16:10.276 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T05:16:10.299 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T05:16:10.306 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T05:16:10.321 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T05:16:10.359 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T05:16:10.562 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T05:16:10.589 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T05:16:10.626 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T05:16:10.628 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T05:16:10.641 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T05:16:10.655 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T05:16:10.682 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T05:16:10.684 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T05:16:10.721 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T05:16:10.931 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T05:16:10.958 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T05:16:10.996 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-paste. 2024-09-13T05:16:10.998 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T05:16:11.022 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T05:16:11.050 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-idna. 2024-09-13T05:16:11.062 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T05:16:11.077 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../049-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T05:16:11.116 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T05:16:11.242 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-portend. 2024-09-13T05:16:11.258 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-09-13T05:16:11.292 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T05:16:11.366 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T05:16:11.369 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T05:16:11.381 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T05:16:11.385 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T05:16:11.416 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T05:16:11.419 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T05:16:11.579 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T05:16:11.605 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T05:16:11.719 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T05:16:11.722 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-requests. 2024-09-13T05:16:11.738 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T05:16:11.746 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T05:16:11.773 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T05:16:11.785 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T05:16:11.812 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T05:16:12.043 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T05:16:12.070 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T05:16:12.097 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T05:16:12.109 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T05:16:12.124 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T05:16:12.140 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T05:16:12.164 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T05:16:12.167 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T05:16:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T05:16:12.404 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T05:16:12.420 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:12.455 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-modules-core (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:12.475 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T05:16:12.502 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T05:16:12.535 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T05:16:12.542 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T05:16:12.562 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T05:16:12.614 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T05:16:12.827 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T05:16:12.854 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:12.889 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T05:16:12.893 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:12.916 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T05:16:12.921 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T05:16:12.947 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T05:16:12.948 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T05:16:12.988 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T05:16:13.205 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T05:16:13.232 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../055-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T05:16:13.242 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-idna. 2024-09-13T05:16:13.269 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../049-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T05:16:13.271 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T05:16:13.291 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T05:16:13.308 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T05:16:13.318 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T05:16:13.357 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T05:16:13.525 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T05:16:13.540 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../056-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T05:16:13.583 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T05:16:13.644 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-mako. 2024-09-13T05:16:13.645 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T05:16:13.660 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T05:16:13.672 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T05:16:13.694 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T05:16:13.711 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T05:16:13.920 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T05:16:13.946 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../057-ceph-mgr_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:13.986 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T05:16:14.032 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-requests. 2024-09-13T05:16:14.057 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T05:16:14.058 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T05:16:14.088 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T05:16:14.123 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T05:16:14.381 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T05:16:14.391 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T05:16:14.408 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../058-ceph-mon_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:14.418 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T05:16:14.435 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T05:16:14.449 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mon (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:14.458 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T05:16:14.462 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T05:16:14.501 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T05:16:14.752 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-webob. 2024-09-13T05:16:14.779 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T05:16:14.780 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T05:16:14.807 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T05:16:14.846 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-modules-core (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:15.027 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T05:16:15.054 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../059-ceph-osd_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:15.093 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-osd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:15.172 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T05:16:15.199 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T05:16:15.242 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T05:16:15.260 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T05:16:15.287 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:15.327 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:15.533 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T05:16:15.560 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T05:16:15.599 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T05:16:15.622 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T05:16:15.649 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../055-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T05:16:15.688 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T05:16:15.886 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-paste. 2024-09-13T05:16:15.913 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T05:16:15.953 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T05:16:16.091 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T05:16:16.119 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../056-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T05:16:16.158 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T05:16:16.272 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph. 2024-09-13T05:16:16.280 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T05:16:16.288 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../060-ceph_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:16.296 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T05:16:16.331 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:16.339 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T05:16:16.503 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T05:16:16.530 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../057-ceph-mgr_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:16.569 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:16.625 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T05:16:16.634 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T05:16:16.652 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T05:16:16.661 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../061-ceph-fuse_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:16.691 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T05:16:16.692 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:16.953 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T05:16:16.980 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T05:16:16.981 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T05:16:17.008 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../058-ceph-mon_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:17.046 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T05:16:17.048 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mon (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:17.073 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../062-ceph-mds_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:17.113 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mds (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:17.136 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T05:16:17.381 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T05:16:17.397 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T05:16:17.439 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T05:16:17.508 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package cephadm. 2024-09-13T05:16:17.535 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../063-cephadm_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:17.574 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:17.643 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T05:16:17.662 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../059-ceph-osd_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:17.701 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-osd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:17.760 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T05:16:17.787 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T05:16:17.827 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T05:16:17.861 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T05:16:17.887 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../064-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T05:16:17.927 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T05:16:18.155 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-idna. 2024-09-13T05:16:18.182 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../049-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T05:16:18.222 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T05:16:18.230 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T05:16:18.246 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../065-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T05:16:18.289 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T05:16:18.541 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T05:16:18.568 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T05:16:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T05:16:18.773 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph. 2024-09-13T05:16:18.799 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../060-ceph_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:18.839 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:18.936 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-requests. 2024-09-13T05:16:18.963 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T05:16:19.011 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T05:16:19.108 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T05:16:19.123 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../061-ceph-fuse_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:19.169 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:19.297 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T05:16:19.324 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T05:16:19.355 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T05:16:19.545 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T05:16:19.572 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../062-ceph-mds_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:19.612 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mds (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:19.684 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T05:16:19.700 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:19.741 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-modules-core (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:20.007 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package cephadm. 2024-09-13T05:16:20.034 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../063-cephadm_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:20.073 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:20.105 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T05:16:20.133 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:20.171 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:20.220 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-babel. 2024-09-13T05:16:20.247 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../066-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T05:16:20.286 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T05:16:20.351 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T05:16:20.367 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../064-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T05:16:20.402 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T05:16:20.491 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T05:16:20.518 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../055-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T05:16:20.557 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T05:16:20.623 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T05:16:20.650 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../067-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T05:16:20.690 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T05:16:20.755 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T05:16:20.781 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../065-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T05:16:20.813 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T05:16:20.860 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T05:16:20.887 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../056-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T05:16:20.927 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T05:16:21.044 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T05:16:21.059 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../068-ceph-mgr-cephadm_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:21.094 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:21.214 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T05:16:21.241 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../057-ceph-mgr_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:21.431 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T05:16:21.458 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T05:16:21.497 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T05:16:21.675 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T05:16:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../058-ceph-mon_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:21.741 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mon (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:21.784 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-py. 2024-09-13T05:16:21.811 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-09-13T05:16:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-babel. 2024-09-13T05:16:21.834 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../066-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T05:16:21.851 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T05:16:21.877 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T05:16:22.155 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T05:16:22.181 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T05:16:22.197 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T05:16:22.221 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T05:16:22.224 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../067-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T05:16:22.263 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T05:16:22.353 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T05:16:22.380 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../059-ceph-osd_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:22.419 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-osd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:22.533 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-toml. 2024-09-13T05:16:22.560 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-09-13T05:16:22.599 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T05:16:22.609 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T05:16:22.635 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../068-ceph-mgr-cephadm_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:22.675 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:22.903 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T05:16:22.929 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T05:16:22.969 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T05:16:23.013 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T05:16:23.039 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T05:16:23.079 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T05:16:23.298 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T05:16:23.325 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T05:16:23.365 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T05:16:23.391 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-py. 2024-09-13T05:16:23.418 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-09-13T05:16:23.457 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T05:16:23.481 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph. 2024-09-13T05:16:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../060-ceph_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:23.540 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:23.685 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-routes. 2024-09-13T05:16:23.712 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T05:16:23.751 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T05:16:23.786 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T05:16:23.813 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T05:16:23.844 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T05:16:23.852 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T05:16:23.870 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../061-ceph-fuse_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:23.909 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:24.089 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T05:16:24.117 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:24.148 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-toml. 2024-09-13T05:16:24.155 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-dashboard (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:24.175 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-09-13T05:16:24.206 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T05:16:24.305 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T05:16:24.332 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../062-ceph-mds_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:24.371 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mds (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:24.509 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T05:16:24.536 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T05:16:24.575 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T05:16:24.732 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephadm. 2024-09-13T05:16:24.759 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../063-cephadm_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:24.790 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:24.938 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T05:16:24.965 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T05:16:25.004 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T05:16:25.068 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T05:16:25.095 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../064-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T05:16:25.134 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T05:16:25.308 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-routes. 2024-09-13T05:16:25.335 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T05:16:25.374 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T05:16:25.378 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T05:16:25.393 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T05:16:25.428 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T05:16:25.437 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T05:16:25.454 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../065-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T05:16:25.487 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T05:16:25.736 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T05:16:25.763 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:25.803 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-dashboard (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:25.882 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T05:16:25.909 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T05:16:25.948 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T05:16:26.319 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T05:16:26.346 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T05:16:26.386 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T05:16:26.551 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-babel. 2024-09-13T05:16:26.578 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../066-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T05:16:26.617 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T05:16:26.689 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T05:16:26.716 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T05:16:26.755 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T05:16:26.954 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T05:16:26.981 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../067-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T05:16:27.020 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T05:16:27.067 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T05:16:27.094 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T05:16:27.133 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T05:16:27.399 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T05:16:27.426 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:27.465 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:27.582 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T05:16:27.609 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../068-ceph-mgr-cephadm_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:27.649 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:27.696 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T05:16:27.723 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T05:16:27.771 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T05:16:27.944 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T05:16:27.960 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T05:16:27.993 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T05:16:28.116 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T05:16:28.143 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T05:16:28.183 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T05:16:28.280 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-py. 2024-09-13T05:16:28.308 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-09-13T05:16:28.347 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T05:16:28.452 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T05:16:28.479 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T05:16:28.494 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T05:16:28.518 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T05:16:28.521 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T05:16:28.561 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T05:16:28.675 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T05:16:28.702 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T05:16:28.741 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T05:16:28.890 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T05:16:28.917 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T05:16:28.948 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T05:16:29.044 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-toml. 2024-09-13T05:16:29.071 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-09-13T05:16:29.111 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T05:16:29.269 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T05:16:29.295 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T05:16:29.296 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T05:16:29.323 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:29.335 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T05:16:29.361 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:29.372 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T05:16:29.399 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T05:16:29.430 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T05:16:29.630 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T05:16:29.657 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-09-13T05:16:29.697 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T05:16:29.767 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T05:16:29.795 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T05:16:29.834 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T05:16:30.010 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T05:16:30.037 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T05:16:30.076 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T05:16:30.146 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-routes. 2024-09-13T05:16:30.173 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T05:16:30.212 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T05:16:30.363 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T05:16:30.379 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T05:16:30.413 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T05:16:30.457 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T05:16:30.484 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T05:16:30.531 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T05:16:30.557 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T05:16:30.584 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:30.623 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-dashboard (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:30.708 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T05:16:30.735 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T05:16:30.774 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T05:16:30.987 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T05:16:31.013 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T05:16:31.061 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T05:16:31.278 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T05:16:31.305 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T05:16:31.344 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T05:16:31.632 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T05:16:31.648 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T05:16:31.682 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T05:16:31.871 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T05:16:31.898 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T05:16:31.929 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T05:16:31.952 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T05:16:31.959 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T05:16:31.975 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:31.979 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-09-13T05:16:32.009 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:32.018 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T05:16:32.329 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T05:16:32.348 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T05:16:32.356 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T05:16:32.375 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T05:16:32.391 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T05:16:32.395 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T05:16:32.414 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T05:16:32.418 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T05:16:32.457 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T05:16:32.707 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T05:16:32.718 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T05:16:32.734 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T05:16:32.745 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T05:16:32.784 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T05:16:32.790 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T05:16:32.828 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T05:16:32.855 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T05:16:32.895 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T05:16:33.002 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package jq. 2024-09-13T05:16:33.029 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T05:16:33.068 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T05:16:33.071 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T05:16:33.098 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T05:16:33.138 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T05:16:33.189 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T05:16:33.216 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T05:16:33.247 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T05:16:33.321 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package socat. 2024-09-13T05:16:33.345 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T05:16:33.388 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T05:16:33.424 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T05:16:33.451 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T05:16:33.491 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T05:16:33.683 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T05:16:33.711 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T05:16:33.750 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T05:16:33.956 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T05:16:33.985 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:34.022 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:34.088 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-test. 2024-09-13T05:16:34.116 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../096-ceph-test_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:34.130 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T05:16:34.146 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:34.154 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-test (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:34.189 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:34.492 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T05:16:34.519 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T05:16:34.978 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T05:16:34.978 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T05:16:34.978 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T05:16:34.978 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T05:16:35.016 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T05:16:35.045 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T05:16:35.083 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T05:16:35.231 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package jq. 2024-09-13T05:16:35.246 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T05:16:35.289 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T05:16:35.446 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T05:16:35.462 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T05:16:35.496 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T05:16:35.559 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package socat. 2024-09-13T05:16:35.586 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T05:16:35.625 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T05:16:35.799 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T05:16:35.827 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T05:16:35.866 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T05:16:35.954 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T05:16:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T05:16:36.012 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T05:16:36.134 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T05:16:36.152 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T05:16:36.161 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../097-ceph-volume_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:36.180 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-09-13T05:16:36.209 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-volume (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:36.218 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T05:16:36.367 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-test. 2024-09-13T05:16:36.382 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../096-ceph-test_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:36.425 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-test (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:36.522 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T05:16:36.539 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T05:16:36.549 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:36.567 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T05:16:36.588 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libcephfs-dev (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:36.606 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T05:16:36.926 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T05:16:36.950 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T05:16:36.956 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T05:16:36.978 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T05:16:36.992 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T05:16:37.008 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T05:16:37.347 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T05:16:37.375 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T05:16:37.412 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T05:16:37.439 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T05:16:37.470 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T05:16:37.531 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T05:16:37.926 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T05:16:37.953 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T05:16:37.973 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T05:16:37.992 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T05:16:38.001 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T05:16:38.040 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T05:16:38.298 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T05:16:38.325 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T05:16:38.364 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T05:16:38.605 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T05:16:38.632 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../097-ceph-volume_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:38.671 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-volume (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:38.717 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T05:16:38.744 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T05:16:38.767 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T05:16:38.784 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T05:16:38.796 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:38.837 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:38.975 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T05:16:39.004 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:39.042 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libcephfs-dev (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:39.156 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T05:16:39.184 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T05:16:39.223 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T05:16:39.230 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T05:16:39.257 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T05:16:39.296 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T05:16:39.387 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T05:16:39.414 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T05:16:39.453 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T05:16:39.534 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T05:16:39.562 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T05:16:39.607 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T05:16:39.609 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T05:16:39.634 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T05:16:39.673 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T05:16:39.801 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T05:16:39.828 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T05:16:39.845 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package jq. 2024-09-13T05:16:39.861 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T05:16:39.867 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T05:16:39.903 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T05:16:39.943 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T05:16:39.970 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T05:16:40.009 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T05:16:40.165 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package socat. 2024-09-13T05:16:40.181 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T05:16:40.223 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T05:16:40.288 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T05:16:40.315 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T05:16:40.362 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T05:16:40.526 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T05:16:40.542 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T05:16:40.584 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T05:16:40.680 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package radosgw. 2024-09-13T05:16:40.683 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T05:16:40.708 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../107-radosgw_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:40.710 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T05:16:40.746 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking radosgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:40.749 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T05:16:40.922 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-test. 2024-09-13T05:16:40.938 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../096-ceph-test_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:40.972 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-test (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:41.086 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T05:16:41.113 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T05:16:41.152 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T05:16:41.501 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T05:16:41.528 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../108-rbd-fuse_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:41.640 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T05:16:41.669 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T05:16:41.706 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T05:16:41.723 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking rbd-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:41.985 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package rbd-nbd. 2024-09-13T05:16:42.001 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../109-rbd-nbd_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:42.043 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking rbd-nbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:42.043 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T05:16:42.070 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T05:16:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T05:16:42.347 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T05:16:42.379 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T05:16:42.406 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T05:16:42.446 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T05:16:42.589 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T05:16:42.715 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T05:16:42.925 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T05:16:42.942 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../097-ceph-volume_18.2.4-820-g48a7de7e-1jammy_all.deb ... 2024-09-13T05:16:42.955 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T05:16:42.975 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-volume (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:43.209 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package radosgw. 2024-09-13T05:16:43.236 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../107-radosgw_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:43.256 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T05:16:43.273 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:43.275 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking radosgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:43.283 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T05:16:43.315 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libcephfs-dev (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:43.693 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T05:16:43.706 INFO:teuthology.orchestra.run.smithi122.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T05:16:43.721 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T05:16:43.759 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T05:16:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:44.003 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T05:16:44.019 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../108-rbd-fuse_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:44.061 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking rbd-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:44.106 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T05:16:44.122 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T05:16:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T05:16:44.275 INFO:teuthology.orchestra.run.smithi122.stdout:Adding system user cephadm....done 2024-09-13T05:16:44.280 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package rbd-nbd. 2024-09-13T05:16:44.296 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../109-rbd-nbd_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:44.330 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking rbd-nbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:44.431 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T05:16:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T05:16:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T05:16:44.659 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T05:16:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T05:16:44.699 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T05:16:44.919 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T05:16:44.934 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T05:16:44.996 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T05:16:45.024 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T05:16:45.037 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T05:16:45.062 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T05:16:45.085 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T05:16:45.285 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T05:16:45.336 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T05:16:45.415 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T05:16:45.443 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T05:16:45.490 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T05:16:45.587 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T05:16:45.607 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T05:16:45.713 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T05:16:45.969 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T05:16:45.997 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T05:16:46.016 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T05:16:46.034 INFO:teuthology.orchestra.run.smithi079.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T05:16:46.035 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T05:16:46.114 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:46.313 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T05:16:46.329 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T05:16:46.363 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T05:16:46.383 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T05:16:46.606 INFO:teuthology.orchestra.run.smithi079.stdout:Adding system user cephadm....done 2024-09-13T05:16:46.632 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T05:16:46.634 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T05:16:46.649 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T05:16:46.716 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T05:16:46.765 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T05:16:46.902 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T05:16:47.041 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T05:16:47.162 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T05:16:47.301 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T05:16:47.419 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T05:16:47.421 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T05:16:47.478 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package radosgw. 2024-09-13T05:16:47.506 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../107-radosgw_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:47.544 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking radosgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:47.653 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T05:16:47.689 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-ceph-argparse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:47.896 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T05:16:47.948 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T05:16:48.017 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T05:16:48.074 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T05:16:48.297 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T05:16:48.314 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../108-rbd-fuse_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:48.316 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T05:16:48.355 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking rbd-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:48.492 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T05:16:48.593 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T05:16:48.617 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package rbd-nbd. 2024-09-13T05:16:48.645 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../109-rbd-nbd_18.2.4-820-g48a7de7e-1jammy_amd64.deb ... 2024-09-13T05:16:48.676 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T05:16:48.683 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking rbd-nbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:48.936 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T05:16:48.961 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T05:16:49.012 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T05:16:49.079 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T05:16:49.204 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T05:16:49.271 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T05:16:49.346 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T05:16:49.389 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T05:16:49.447 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T05:16:49.606 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T05:16:49.657 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T05:16:49.715 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T05:16:49.907 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T05:16:49.966 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-ceph-argparse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:49.970 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T05:16:50.234 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T05:16:50.360 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T05:16:50.396 INFO:teuthology.orchestra.run.smithi016.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T05:16:50.458 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:50.575 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T05:16:50.611 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T05:16:50.712 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T05:16:50.827 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T05:16:50.952 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T05:16:50.999 INFO:teuthology.orchestra.run.smithi016.stdout:Adding system user cephadm....done 2024-09-13T05:16:51.055 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T05:16:51.079 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T05:16:51.156 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T05:16:51.189 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T05:16:51.388 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T05:16:51.424 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T05:16:51.466 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T05:16:51.650 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T05:16:51.733 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T05:16:51.773 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T05:16:51.776 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T05:16:52.024 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T05:16:52.035 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T05:16:52.035 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T05:16:52.295 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T05:16:52.392 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T05:16:52.421 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T05:16:52.654 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T05:16:52.659 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T05:16:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T05:16:52.995 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T05:16:53.055 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T05:16:53.099 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T05:16:53.189 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T05:16:53.276 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T05:16:53.315 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T05:16:53.405 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:53.582 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T05:16:53.625 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T05:16:53.698 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T05:16:53.910 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T05:16:54.010 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T05:16:54.024 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T05:16:54.169 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T05:16:54.244 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T05:16:54.421 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T05:16:54.596 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T05:16:54.672 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-ceph-argparse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:54.906 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T05:16:54.981 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T05:16:55.107 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T05:16:55.188 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:16:55.249 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T05:16:55.322 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T05:16:55.350 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T05:16:55.451 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T05:16:55.525 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T05:16:55.582 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T05:16:55.634 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:16:55.708 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T05:16:55.810 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T05:16:55.834 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T05:16:55.902 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T05:16:55.936 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T05:16:56.127 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T05:16:56.204 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T05:16:56.221 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T05:16:56.386 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T05:16:56.497 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T05:16:56.637 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:16:56.763 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-13T05:16:56.823 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T05:16:56.889 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T05:16:57.374 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T05:16:57.401 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:16:57.511 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T05:16:57.525 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T05:16:57.633 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T05:16:57.762 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T05:16:57.776 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T05:16:57.889 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T05:16:57.893 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T05:16:57.902 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T05:16:58.015 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T05:16:58.028 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T05:16:58.152 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T05:16:58.316 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T05:16:58.329 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T05:16:58.403 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T05:16:58.576 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T05:16:58.663 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T05:16:58.787 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T05:16:58.789 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T05:16:58.818 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:16:58.945 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-13T05:16:59.065 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T05:16:59.065 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T05:16:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T05:16:59.183 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T05:16:59.381 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T05:16:59.450 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T05:16:59.451 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T05:16:59.641 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T05:16:59.735 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T05:16:59.751 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T05:16:59.900 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T05:17:00.027 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T05:17:00.094 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T05:17:00.176 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T05:17:00.337 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T05:17:00.402 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T05:17:00.411 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T05:17:00.463 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:00.647 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T05:17:00.653 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T05:17:00.756 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T05:17:00.771 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T05:17:00.773 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T05:17:01.056 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T05:17:01.090 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T05:17:01.158 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T05:17:01.182 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T05:17:01.409 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T05:17:01.500 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T05:17:01.693 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:17:01.793 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T05:17:01.819 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T05:17:02.053 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T05:17:02.137 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T05:17:02.288 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:17:02.409 INFO:teuthology.orchestra.run.smithi079.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T05:17:02.422 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T05:17:02.439 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T05:17:02.664 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T05:17:02.681 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T05:17:02.715 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T05:17:02.790 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T05:17:02.829 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T05:17:02.840 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T05:17:02.959 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T05:17:03.100 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T05:17:03.200 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T05:17:03.277 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T05:17:03.351 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T05:17:03.443 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T05:17:03.545 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T05:17:03.610 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T05:17:03.711 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:17:03.804 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:17:03.829 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T05:17:03.930 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-13T05:17:04.020 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T05:17:04.056 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T05:17:04.164 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T05:17:04.372 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T05:17:04.383 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T05:17:04.456 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T05:17:04.634 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T05:17:04.707 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T05:17:04.724 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T05:17:04.808 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librados2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:04.842 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T05:17:04.893 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T05:17:04.909 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librgw2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:05.018 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libsqlite3-mod-ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:05.085 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T05:17:05.144 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libcephfs2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:05.152 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T05:17:05.279 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libradosstriper1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:05.328 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T05:17:05.387 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T05:17:05.490 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T05:17:05.637 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T05:17:05.654 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T05:17:05.783 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librbd1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:05.805 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T05:17:05.909 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-modules-core (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:06.035 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:06.047 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T05:17:06.115 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T05:17:06.225 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T05:17:06.258 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T05:17:06.258 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T05:17:06.340 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T05:17:06.501 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T05:17:06.676 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T05:17:06.737 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libcephfs-dev (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:06.794 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librados2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:06.835 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T05:17:06.846 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rados (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:06.920 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librgw2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:06.972 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T05:17:07.046 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libsqlite3-mod-ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:07.095 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T05:17:07.197 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libcephfs2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:07.323 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libradosstriper1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:07.477 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T05:17:07.519 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T05:17:07.739 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T05:17:07.795 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librbd1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:07.874 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T05:17:07.930 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-modules-core (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:08.056 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:08.185 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:08.258 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T05:17:08.311 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up rbd-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:08.420 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up rbd-nbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:08.509 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T05:17:08.530 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:08.536 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T05:17:08.536 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T05:17:08.656 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cephfs (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:08.777 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T05:17:08.782 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:08.878 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T05:17:09.007 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libcephfs-dev (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:09.134 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rados (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:09.196 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T05:17:09.243 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T05:17:09.316 INFO:teuthology.orchestra.run.smithi122.stdout:Adding group ceph....done 2024-09-13T05:17:09.497 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T05:17:09.693 INFO:teuthology.orchestra.run.smithi122.stdout:Adding system user ceph....done 2024-09-13T05:17:09.790 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T05:17:09.861 INFO:teuthology.orchestra.run.smithi122.stdout:Setting system user ceph properties....done 2024-09-13T05:17:09.877 INFO:teuthology.orchestra.run.smithi122.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T05:17:09.899 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T05:17:10.017 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T05:17:10.184 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T05:17:10.364 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T05:17:10.443 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T05:17:10.481 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:10.608 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up rbd-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:10.702 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T05:17:10.750 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up rbd-nbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:10.825 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-test (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:10.851 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:10.951 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up radosgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:10.952 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cephfs (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:11.070 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:11.087 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T05:17:11.372 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T05:17:11.418 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T05:17:11.418 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T05:17:11.622 INFO:teuthology.orchestra.run.smithi079.stdout:Adding group ceph....done 2024-09-13T05:17:11.716 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T05:17:11.817 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librados2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:11.877 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-base (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:11.918 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librgw2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:12.024 INFO:teuthology.orchestra.run.smithi079.stdout:Adding system user ceph....done 2024-09-13T05:17:12.044 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libsqlite3-mod-ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:12.170 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libcephfs2 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:12.184 INFO:teuthology.orchestra.run.smithi079.stdout:Setting system user ceph properties....done 2024-09-13T05:17:12.198 INFO:teuthology.orchestra.run.smithi079.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T05:17:12.230 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T05:17:12.296 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libradosstriper1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:12.329 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T05:17:12.507 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T05:17:12.635 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T05:17:12.671 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mds (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:12.775 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librbd1 (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:12.896 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T05:17:12.896 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T05:17:12.902 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-modules-core (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:13.011 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:13.114 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-test (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:13.213 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T05:17:13.214 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T05:17:13.240 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up radosgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:13.389 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:13.620 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T05:17:13.620 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T05:17:13.671 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libcephfs-dev (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:13.717 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T05:17:13.717 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T05:17:13.797 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rados (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:13.923 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T05:17:14.108 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-osd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:14.209 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-base (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:14.351 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T05:17:14.351 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T05:17:14.584 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T05:17:14.842 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-k8sevents (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:14.968 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.053 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mds (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mon (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.228 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.297 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T05:17:15.297 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T05:17:15.317 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T05:17:15.318 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T05:17:15.353 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up rbd-fuse (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.479 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up rbd-nbd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.605 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.731 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cephfs (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.789 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.812 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.858 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-common (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:15.955 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:16.002 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T05:17:16.002 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T05:17:16.081 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-dashboard (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:16.232 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-volume (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:16.317 INFO:teuthology.orchestra.run.smithi016.stdout:Adding group ceph....done 2024-09-13T05:17:16.491 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-osd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:16.576 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T05:17:16.651 INFO:teuthology.orchestra.run.smithi016.stdout:Adding system user ceph....done 2024-09-13T05:17:16.745 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T05:17:16.745 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T05:17:16.813 INFO:teuthology.orchestra.run.smithi016.stdout:Setting system user ceph properties....done 2024-09-13T05:17:16.826 INFO:teuthology.orchestra.run.smithi016.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T05:17:16.970 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T05:17:17.219 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-k8sevents (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:17.301 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T05:17:17.413 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:17.660 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mon (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:17.686 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T05:17:17.751 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-test (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:17.877 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up radosgw (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:17.917 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T05:17:17.917 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T05:17:18.188 INFO:teuthology.orchestra.run.smithi122.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T05:17:18.194 DEBUG:teuthology.parallel:result is None 2024-09-13T05:17:18.314 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T05:17:18.315 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T05:17:18.432 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:18.575 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:18.745 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-base (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:18.870 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-dashboard (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:19.021 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-volume (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:19.085 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T05:17:19.374 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T05:17:19.542 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mds (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:19.814 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T05:17:19.814 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T05:17:20.143 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T05:17:20.301 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:20.586 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T05:17:20.586 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T05:17:20.621 INFO:teuthology.orchestra.run.smithi079.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T05:17:20.628 DEBUG:teuthology.parallel:result is None 2024-09-13T05:17:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-osd (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T05:17:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T05:17:21.762 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-k8sevents (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:21.896 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mon (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:22.216 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T05:17:22.216 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T05:17:22.732 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-cephadm (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:22.892 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:23.377 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-dashboard (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:23.528 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-volume (18.2.4-820-g48a7de7e-1jammy) ... 2024-09-13T05:17:23.922 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T05:17:24.681 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T05:17:25.166 INFO:teuthology.orchestra.run.smithi016.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T05:17:25.170 DEBUG:teuthology.parallel:result is None 2024-09-13T05:17:25.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T05:17:25.322 DEBUG:teuthology.orchestra.run.smithi016:> dpkg-query -W -f '${Version}' ceph 2024-09-13T05:17:25.341 INFO:teuthology.orchestra.run.smithi016.stdout:18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:17:25.342 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:17:25.342 INFO:teuthology.task.install:The correct ceph version 18.2.4-820-g48a7de7e-1jammy is installed. 2024-09-13T05:17:25.344 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T05:17:25.493 DEBUG:teuthology.orchestra.run.smithi079:> dpkg-query -W -f '${Version}' ceph 2024-09-13T05:17:25.513 INFO:teuthology.orchestra.run.smithi079.stdout:18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:17:25.514 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:17:25.514 INFO:teuthology.task.install:The correct ceph version 18.2.4-820-g48a7de7e-1jammy is installed. 2024-09-13T05:17:25.516 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T05:17:25.666 DEBUG:teuthology.orchestra.run.smithi122:> dpkg-query -W -f '${Version}' ceph 2024-09-13T05:17:25.685 INFO:teuthology.orchestra.run.smithi122.stdout:18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:17:25.685 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-820-g48a7de7e-1jammy 2024-09-13T05:17:25.685 INFO:teuthology.task.install:The correct ceph version 18.2.4-820-g48a7de7e-1jammy is installed. 2024-09-13T05:17:25.687 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T05:17:25.687 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:25.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T05:17:25.702 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:25.703 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T05:17:25.716 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:17:25.716 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T05:17:25.741 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T05:17:25.741 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:25.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T05:17:25.757 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T05:17:25.813 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:25.814 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T05:17:25.827 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T05:17:25.882 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:17:25.882 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T05:17:25.896 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T05:17:25.951 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T05:17:25.951 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:25.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T05:17:25.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T05:17:26.021 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:26.021 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T05:17:26.035 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T05:17:26.090 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:17:26.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T05:17:26.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T05:17:26.159 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T05:17:26.160 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:26.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T05:17:26.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T05:17:26.229 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:26.229 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T05:17:26.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T05:17:26.298 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:17:26.298 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T05:17:26.312 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T05:17:26.366 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T05:17:26.445 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T05:17:26.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2024-09-13T05:17:26.447 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /var/log/ceph 2024-09-13T05:17:26.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-09-13T05:17:26.462 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T05:17:26.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T05:17:26.503 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T05:17:26.505 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T05:17:26.517 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T05:17:26.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T05:17:26.559 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T05:17:26.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T05:17:26.583 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T05:17:26.583 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T05:17:26.583 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857', 'branch': 'reef', 'description': 'rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_nbd}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900857', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '7a853b1706d076d58e0da7a5d3b9b9eb36276880'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 248, 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412', 'sleep_before_teardown': 0, 'subset': '116/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7a853b1706d076d58e0da7a5d3b9b9eb36276880', 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJZmayn87lprQi+8OG7heq7Wj+WJEQypxXjSR2AqE6F5b4iR4OujIELR/3fcjG36j6n3aH5fNrDNlXtV8X4H7mU=', 'smithi079.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAV69TgMtkN7Iao6DM43kZ2fCYq4JRdLAiOMezOg4x9420UWQYH4z4ccvxA7guUA7vJGffrD+TMeuGR4gEbq1LI=', 'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBANUGUAfyNPu08llXue3k7KGxoL4/ERtW75hSIumivEzK5FuEea4KjfZdLSXXzDT/hoqAenSuBO8UbWmWRNHBY8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/rbd-nbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_22:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T05:17:26.584 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T05:17:26.615 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T05:17:26.617 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T05:17:26.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T05:17:26.663 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T05:17:26.665 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T05:17:26.680 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:26.680 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T05:17:26.722 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T05:17:26.722 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-09-13T05:17:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T05:17:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T05:17:26.771 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-09-13T05:17:26.771 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T05:17:26.771 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-13 05:11:57.550027513 +0000 2024-09-13T05:17:26.771 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-13 05:11:57.546027548 +0000 2024-09-13T05:17:26.771 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-13 05:11:57.546027548 +0000 2024-09-13T05:17:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-09-13T05:17:26.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T05:17:26.824 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-13T05:17:26.825 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-13T05:17:26.825 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000308923 s, 1.7 MB/s 2024-09-13T05:17:26.826 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T05:17:26.876 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-09-13T05:17:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T05:17:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T05:17:26.922 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-09-13T05:17:26.922 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T05:17:26.922 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-13 05:11:58.350020367 +0000 2024-09-13T05:17:26.923 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-13 05:11:58.082022760 +0000 2024-09-13T05:17:26.923 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-13 05:11:58.082022760 +0000 2024-09-13T05:17:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-09-13T05:17:26.923 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T05:17:26.976 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-13T05:17:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-13T05:17:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000324391 s, 1.6 MB/s 2024-09-13T05:17:26.978 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T05:17:27.028 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-09-13T05:17:27.074 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T05:17:27.074 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T05:17:27.074 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-09-13T05:17:27.074 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T05:17:27.074 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-13 05:11:58.798016365 +0000 2024-09-13T05:17:27.075 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-13 05:11:58.558018509 +0000 2024-09-13T05:17:27.075 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-13 05:11:58.558018509 +0000 2024-09-13T05:17:27.075 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-09-13T05:17:27.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T05:17:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-13T05:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-13T05:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000289014 s, 1.8 MB/s 2024-09-13T05:17:27.130 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T05:17:27.183 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-09-13T05:17:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T05:17:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T05:17:27.230 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 935 Links: 1 2024-09-13T05:17:27.230 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T05:17:27.230 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-13 05:11:59.266012185 +0000 2024-09-13T05:17:27.231 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-13 05:11:59.006014507 +0000 2024-09-13T05:17:27.231 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-13 05:11:59.006014507 +0000 2024-09-13T05:17:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-09-13T05:17:27.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T05:17:27.284 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-13T05:17:27.284 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-13T05:17:27.284 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000194421 s, 2.6 MB/s 2024-09-13T05:17:27.285 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T05:17:27.336 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T05:17:27.336 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:27.336 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T05:17:27.341 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T05:17:27.342 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-09-13T05:17:27.387 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T05:17:27.387 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T05:17:27.388 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-09-13T05:17:27.388 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T05:17:27.388 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-13 05:11:55.267562409 +0000 2024-09-13T05:17:27.388 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-13 05:11:55.043564408 +0000 2024-09-13T05:17:27.388 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-13 05:11:55.043564408 +0000 2024-09-13T05:17:27.388 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2024-09-13T05:17:27.388 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T05:17:27.442 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-13T05:17:27.442 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-13T05:17:27.442 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000434433 s, 1.2 MB/s 2024-09-13T05:17:27.444 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T05:17:27.497 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-09-13T05:17:27.543 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T05:17:27.543 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T05:17:27.543 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-09-13T05:17:27.543 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T05:17:27.543 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-13 05:11:55.743558156 +0000 2024-09-13T05:17:27.543 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-13 05:11:55.491560408 +0000 2024-09-13T05:17:27.543 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-13 05:11:55.491560408 +0000 2024-09-13T05:17:27.544 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2024-09-13T05:17:27.544 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T05:17:27.598 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-13T05:17:27.598 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-13T05:17:27.598 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000449123 s, 1.1 MB/s 2024-09-13T05:17:27.599 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T05:17:27.648 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-09-13T05:17:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T05:17:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T05:17:27.695 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-09-13T05:17:27.695 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T05:17:27.695 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-13 05:11:56.231553797 +0000 2024-09-13T05:17:27.695 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-13 05:11:55.967556155 +0000 2024-09-13T05:17:27.695 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-13 05:11:55.967556155 +0000 2024-09-13T05:17:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2024-09-13T05:17:27.695 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T05:17:27.749 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-13T05:17:27.749 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-13T05:17:27.749 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000417424 s, 1.2 MB/s 2024-09-13T05:17:27.750 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T05:17:27.800 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-09-13T05:17:27.846 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T05:17:27.847 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T05:17:27.847 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 933 Links: 1 2024-09-13T05:17:27.847 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T05:17:27.847 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-13 05:11:56.731549332 +0000 2024-09-13T05:17:27.847 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-13 05:11:56.471551654 +0000 2024-09-13T05:17:27.847 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-13 05:11:56.471551654 +0000 2024-09-13T05:17:27.847 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2024-09-13T05:17:27.847 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T05:17:27.901 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-13T05:17:27.902 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-13T05:17:27.902 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000380787 s, 1.3 MB/s 2024-09-13T05:17:27.903 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T05:17:27.952 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T05:17:27.952 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-13T05:17:27.952 INFO:tasks.ceph:Generating config... 2024-09-13T05:17:27.954 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T05:17:27.954 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-13T05:17:27.954 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-13T05:17:27.954 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T05:17:27.954 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T05:17:27.954 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T05:17:27.955 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T05:17:27.955 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T05:17:27.955 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T05:17:27.956 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-13T05:17:27.956 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T05:17:27.956 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-13T05:17:27.956 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-13T05:17:27.956 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T05:17:27.956 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T05:17:27.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T05:17:27.991 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T05:17:28.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T05:17:28.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T05:17:28.145 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.16'), ('mon.c', '[v2:172.21.15.16:3301,v1:172.21.15.16:6790]'), ('mon.b', '172.21.15.79')] 2024-09-13T05:17:28.146 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.79', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-13T05:17:28.146 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:28.146 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T05:17:28.194 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.16 --addv c '[v2:172.21.15.16:3301,v1:172.21.15.16:6790]' --add b 172.21.15.79 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid 1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = pacific 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2024-09-13T05:17:28.281515+0000 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-13T05:17:28.281515+0000 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 16 (pacific) 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:election_strategy: 1 2024-09-13T05:17:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-13T05:17:28.282 INFO:teuthology.orchestra.run.smithi016.stdout:1: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.b 2024-09-13T05:17:28.282 INFO:teuthology.orchestra.run.smithi016.stdout:2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-09-13T05:17:28.282 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T05:17:28.283 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T05:17:28.330 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5... 2024-09-13T05:17:28.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T05:17:28.375 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T05:17:28.377 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T05:17:28.405 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-09-13T05:17:28.405 INFO:teuthology.orchestra.run.smithi079.stdout: chdir = "" 2024-09-13T05:17:28.405 INFO:teuthology.orchestra.run.smithi079.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T05:17:28.405 INFO:teuthology.orchestra.run.smithi079.stdout: auth supported = cephx 2024-09-13T05:17:28.405 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.405 INFO:teuthology.orchestra.run.smithi079.stdout: filestore xattr use omap = true 2024-09-13T05:17:28.405 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: mon clock drift allowed = 1.000 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: osd crush chooseleaf type = 0 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: auth debug = true 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on old message = true 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on bug = true 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: mon pg warn max object skew = 0 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T05:17:28.406 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default size = 2 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow primary affinity = true 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow pg remap = true 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on legacy crush tunables = false 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on crush straw calc version zero = false 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on no sortbitwise = false 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on osd down out interval zero = false 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on too few osds = false 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T05:17:28.407 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: mon_allow_pool_size_one = true 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: osd default data pool replay window = 5 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: mon allow pool delete = true 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: mon cluster log file level = debug 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: debug asserts on shutdown = true 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: mon health detail to clog = false 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: mon host = "172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.79" 2024-09-13T05:17:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: mon client directed command retry = 5 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on pool no app = False 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: ms inject socket failures = 5000 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout:[osd] 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: osd journal size = 100 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub load threshold = 5.0 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub max interval = 600 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock profile = high_recovery_ops 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: osd recover clone overlap = true 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: osd recovery max chunk = 1048576 2024-09-13T05:17:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug shutdown = true 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug op order = true 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify stray on activate = true 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: osd open classes on start = true 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug pg log writeout = true 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: osd deep scrub update digest min age = 30 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: osd map max advance = 10 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: journal zero on create = true 2024-09-13T05:17:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: filestore ondisk finisher threads = 3 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: filestore apply finisher threads = 3 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: bdev debug aio = true 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug misdirected ops = true 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: bdev async discard = True 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: bdev enable discard = True 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore block size = 96636764160 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore fsck on mount = True 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluefs = 1/20 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluestore = 1/20 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: debug osd = 20 2024-09-13T05:17:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: debug rocksdb = 4/10 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd full ratio = 0.9 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd nearfull ratio = 0.8 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: osd blocked scrub grace period = 3600 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: osd failsafe full ratio = 0.95 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: osd max markdown count = 1000 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: osd memory target = 1610612736 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: osd objectstore = bluestore 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr] 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: debug mgr = 20 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-09-13T05:17:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: mgr/telemetry/nag = false 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout:[mon] 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: debug paxos = 20 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: mon data avail warn = 5 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: mon mgr mkfs grace = 240 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-09-13T05:17:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd reporter subtree level = osd 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd prime pg temp = true 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: # 1m isn't quite enough 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: mon_down_mkfs_grace = 2m 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_filestore_osds = false 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout:[client] 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: rgw cache enabled = true 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable ops log = true 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable usage log = true 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.a] 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.c] 2024-09-13T05:17:28.415 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.b] 2024-09-13T05:17:28.416 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-09-13T05:17:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-09-13T05:17:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T05:17:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-09-13T05:17:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-09-13T05:17:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T05:17:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.79" 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on pool no app = False 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5 2024-09-13T05:17:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-09-13T05:17:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-09-13T05:17:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: osd blocked scrub grace period = 3600 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: osd max markdown count = 1000 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: osd memory target = 1610612736 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-09-13T05:17:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T05:17:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-09-13T05:17:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-09-13T05:17:28.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T05:17:28.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-09-13T05:17:28.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.79" 2024-09-13T05:17:28.429 INFO:teuthology.orchestra.run.smithi016.stdout: mon client directed command retry = 5 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on pool no app = False 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject socket failures = 5000 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-09-13T05:17:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.431 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1/20 2024-09-13T05:17:28.432 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1/20 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 4/10 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: osd blocked scrub grace period = 3600 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: osd max markdown count = 1000 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: osd memory target = 1610612736 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-09-13T05:17:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-09-13T05:17:28.434 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-09-13T05:17:28.435 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-09-13T05:17:28.436 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-09-13T05:17:28.438 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T05:17:28.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T05:17:28.531 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T05:17:28.531 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:28.532 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T05:17:28.537 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:28.537 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T05:17:28.586 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2024-09-13T05:17:28.587 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:28.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T05:17:28.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T05:17:28.651 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:28.651 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:28.698 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:17:28.699 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:28.699 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T05:17:28.699 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T05:17:28.722 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:28.722 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:28.767 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T05:17:28.767 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:17:28.767 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T05:17:28.768 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T05:17:28.791 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:17:28.791 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:28.840 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T05:17:28.840 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T05:17:28.840 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T05:17:28.883 INFO:teuthology.orchestra.run.smithi079.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T05:17:28.893 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T05:17:28.893 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T05:17:28.894 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T05:17:28.928 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T05:17:28.948 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T05:17:28.948 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-13T05:17:28.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T05:17:28.962 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T05:17:28.963 INFO:tasks.ceph:role: osd.0 2024-09-13T05:17:28.963 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-13T05:17:28.963 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T05:17:29.018 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T05:17:29.019 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T05:17:29.019 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T05:17:29.019 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T05:17:29.019 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T05:17:29.019 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-13T05:17:29.019 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T05:17:29.019 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T05:17:29.019 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T05:17:29.019 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T05:17:29.044 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-13T05:17:29.045 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-13T05:17:29.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T05:17:29.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T05:17:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:17:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T05:17:29.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T05:17:29.208 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T05:17:29.208 INFO:tasks.ceph:role: osd.1 2024-09-13T05:17:29.208 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-13T05:17:29.208 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T05:17:29.261 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T05:17:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T05:17:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T05:17:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T05:17:29.262 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T05:17:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-13T05:17:29.262 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T05:17:29.262 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T05:17:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T05:17:29.262 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T05:17:29.289 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-13T05:17:29.290 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-13T05:17:29.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-13T05:17:29.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T05:17:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:17:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T05:17:29.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T05:17:29.457 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T05:17:29.457 INFO:tasks.ceph:role: osd.2 2024-09-13T05:17:29.457 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-13T05:17:29.457 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T05:17:29.510 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T05:17:29.536 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-13T05:17:29.538 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-13T05:17:29.538 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-13T05:17:29.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T05:17:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:17:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T05:17:29.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:29.768+0000 7f901d2637c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T05:17:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:29.768+0000 7f901d2637c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T05:17:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:29.768+0000 7f901d2637c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T05:17:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:29.768+0000 7f901d2637c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T05:17:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:29.768+0000 7f901d2637c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T05:17:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:29.772+0000 7f901d2637c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T05:17:32.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T05:17:32.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:32.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:32.836+0000 7fd1f14927c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T05:17:32.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:32.840+0000 7fd1f14927c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T05:17:32.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:32.840+0000 7fd1f14927c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T05:17:32.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:32.840+0000 7fd1f14927c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T05:17:32.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:32.840+0000 7fd1f14927c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T05:17:32.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:32.840+0000 7fd1f14927c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T05:17:35.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T05:17:35.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:35.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:35.968+0000 7fcc06e5b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T05:17:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:35.968+0000 7fcc06e5b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T05:17:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:35.968+0000 7fcc06e5b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T05:17:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:35.968+0000 7fcc06e5b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T05:17:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:35.968+0000 7fcc06e5b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T05:17:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-13T05:17:35.968+0000 7fcc06e5b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T05:17:38.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T05:17:39.006 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T05:17:39.020 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T05:17:39.021 INFO:tasks.ceph:role: osd.3 2024-09-13T05:17:39.021 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:17:39.021 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T05:17:39.081 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T05:17:39.081 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T05:17:39.082 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T05:17:39.082 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T05:17:39.082 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T05:17:39.082 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-09-13T05:17:39.082 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T05:17:39.082 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T05:17:39.082 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T05:17:39.082 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T05:17:39.564 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-09-13T05:17:39.566 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-09-13T05:17:39.567 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-13T05:17:39.720 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T05:17:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T05:17:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:17:39.733 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T05:17:39.786 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T05:17:39.786 INFO:tasks.ceph:role: osd.4 2024-09-13T05:17:39.787 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:17:39.787 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T05:17:39.850 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T05:17:39.850 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T05:17:39.851 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T05:17:39.851 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T05:17:39.851 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T05:17:39.851 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-09-13T05:17:39.851 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T05:17:39.851 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T05:17:39.851 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T05:17:39.851 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T05:17:40.298 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-09-13T05:17:40.300 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-09-13T05:17:40.300 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-13T05:17:40.394 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T05:17:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:17:40.406 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T05:17:40.406 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T05:17:40.463 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T05:17:40.463 INFO:tasks.ceph:role: osd.5 2024-09-13T05:17:40.463 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:17:40.463 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T05:17:40.526 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T05:17:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T05:17:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T05:17:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T05:17:40.527 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T05:17:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-09-13T05:17:40.527 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T05:17:40.527 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T05:17:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T05:17:40.527 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T05:17:40.983 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-09-13T05:17:40.985 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-09-13T05:17:40.986 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-13T05:17:41.092 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T05:17:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:17:41.105 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T05:17:41.106 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:41.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:41.209+0000 7f1580d2b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T05:17:41.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:41.213+0000 7f1580d2b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T05:17:41.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:41.213+0000 7f1580d2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T05:17:41.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:41.213+0000 7f1580d2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T05:17:41.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:41.213+0000 7f1580d2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T05:17:41.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:41.221+0000 7f1580d2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T05:17:44.194 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T05:17:44.207 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:44.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:44.288+0000 7f8829e3d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T05:17:44.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:44.288+0000 7f8829e3d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T05:17:44.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:44.288+0000 7f8829e3d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T05:17:44.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:44.288+0000 7f8829e3d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T05:17:44.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:44.288+0000 7f8829e3d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T05:17:44.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:44.288+0000 7f8829e3d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T05:17:47.277 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T05:17:47.331 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:47.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:47.416+0000 7fc1827c27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T05:17:47.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:47.416+0000 7fc1827c27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T05:17:47.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:47.416+0000 7fc1827c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T05:17:47.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:47.416+0000 7fc1827c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T05:17:47.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:47.416+0000 7fc1827c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T05:17:47.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-09-13T05:17:47.420+0000 7fc1827c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T05:17:50.416 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T05:17:50.472 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T05:17:50.472 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:50.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T05:17:50.487 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:50.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T05:17:50.542 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:50.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T05:17:50.598 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:50.598 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T05:17:50.612 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:50.612 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T05:17:50.666 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:50.666 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T05:17:50.717 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:50.721 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T05:17:50.770 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:17:50.770 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T05:17:50.777 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T05:17:50.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T05:17:50.779 DEBUG:teuthology.orchestra.run.smithi079:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T05:17:50.816 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDpyuNmPfYfLhAAUNoYyTXMiBVqMtGgc/UGAw== 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDsyuNmPmQyMhAA5karBsgK1SAxPQPy3f1Dow== 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDvyuNms7PfORAAacGd9DX373NFElbBnVJW/w== 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDoyuNmI3PINBAAt5rRfIfHfW/SSGjGWQFXzg== 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQD1yuNmRpWNDBAAJIuKbl60rdgTgXeaORTOtg== 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.4] 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQD4yuNmZfpcERAAy0FxoFc5flENuQOuPqaSdg== 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.5] 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQD7yuNmuRAMGRAA2yBo/rKEGJY6KFVYZYS0eQ== 2024-09-13T05:17:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-09-13T05:17:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDoyuNm4ER9NxAAYmglYFKT693xNJbu+Pk8Kg== 2024-09-13T05:17:50.825 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2024-09-13T05:17:50.825 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDpyuNmPfYfLhAAUNoYyTXMiBVqMtGgc/UGAw== 2024-09-13T05:17:50.825 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2024-09-13T05:17:50.825 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDsyuNmPmQyMhAA5karBsgK1SAxPQPy3f1Dow== 2024-09-13T05:17:50.825 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDvyuNms7PfORAAacGd9DX373NFElbBnVJW/w== 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.x] 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDoyuNmI3PINBAAt5rRfIfHfW/SSGjGWQFXzg== 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.3] 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQD1yuNmRpWNDBAAJIuKbl60rdgTgXeaORTOtg== 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.4] 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQD4yuNmZfpcERAAy0FxoFc5flENuQOuPqaSdg== 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.5] 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQD7yuNmuRAMGRAA2yBo/rKEGJY6KFVYZYS0eQ== 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-09-13T05:17:50.826 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDoyuNm4ER9NxAAYmglYFKT693xNJbu+Pk8Kg== 2024-09-13T05:17:50.827 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:50.863 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:50.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:50.918 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:50.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:51.003 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:51.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T05:17:51.087 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T05:17:51.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:51.133 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:51.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:51.219 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:51.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:51.307 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T05:17:51.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T05:17:51.391 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T05:17:51.466 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T05:17:51.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T05:17:51.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T05:17:51.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T05:17:51.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T05:17:51.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T05:17:51.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T05:17:51.740 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T05:17:51.755 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T05:17:51.853 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T05:17:51.866 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:51.869 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T05:17:51.914 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T05:17:51.914 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T05:17:51.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T05:17:51.916 INFO:tasks.ceph.mon.a:Started 2024-09-13T05:17:51.916 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T05:17:51.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T05:17:51.919 INFO:tasks.ceph.mon.c:Started 2024-09-13T05:17:51.919 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T05:17:51.919 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T05:17:51.959 INFO:tasks.ceph.mon.b:Started 2024-09-13T05:17:51.960 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T05:17:51.960 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T05:17:51.960 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T05:17:51.962 INFO:tasks.ceph.mgr.x:Started 2024-09-13T05:17:51.962 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T05:17:51.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T05:17:57.617 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:57.616+0000 7f9baf137180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T05:17:58.138 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:58.136+0000 7f9baf137180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T05:17:58.191 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-09-13T05:17:58.211 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T05:17:58.211 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T05:17:58.211 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:58.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T05:17:58.225 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:58.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T05:17:58.281 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-13T05:17:58.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T05:17:58.342 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:58.342 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T05:17:58.356 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:58.356 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T05:17:58.356 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:58.356+0000 7f9baf137180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T05:17:58.368 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-13T05:17:58.368 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T05:17:58.421 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new e58174f4-85ec-4321-8bc3-384261c59c6b 0 2024-09-13T05:17:58.679 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-09-13T05:17:58.698 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new 84a3db6c-dacd-43d5-b964-37fdef0091a4 1 2024-09-13T05:17:58.901 INFO:teuthology.orchestra.run.smithi079.stdout:1 2024-09-13T05:17:58.918 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new 0d530476-cf38-4544-9ce7-e8ed35df220b 2 2024-09-13T05:17:58.943 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:58.944+0000 7f9baf137180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T05:17:58.999 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:59.000+0000 7f9baf137180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T05:17:59.053 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:59.052+0000 7f9baf137180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T05:17:59.112 INFO:teuthology.orchestra.run.smithi079.stdout:2 2024-09-13T05:17:59.128 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new fd91f7e2-580d-47ea-9db5-792c8f1ebe01 3 2024-09-13T05:17:59.135 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:59.136+0000 7f9baf137180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T05:17:59.192 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:59.192+0000 7f9baf137180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T05:17:59.276 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:59.276+0000 7f9baf137180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T05:17:59.319 INFO:teuthology.orchestra.run.smithi079.stdout:3 2024-09-13T05:17:59.332 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:59.332+0000 7f9baf137180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T05:17:59.336 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new 01af0f4c-5da2-46fe-a262-59fcc6a2efa9 4 2024-09-13T05:17:59.488 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:59.488+0000 7f9baf137180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T05:17:59.528 INFO:teuthology.orchestra.run.smithi079.stdout:4 2024-09-13T05:17:59.545 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new 64cdd5f2-1568-4e96-ad90-6ef7414651f0 5 2024-09-13T05:17:59.550 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:17:59.552+0000 7f9baf137180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T05:17:59.746 INFO:teuthology.orchestra.run.smithi079.stdout:5 2024-09-13T05:17:59.762 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T05:17:59.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T05:17:59.765 INFO:tasks.ceph.osd.0:Started 2024-09-13T05:17:59.765 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T05:17:59.765 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T05:17:59.767 INFO:tasks.ceph.osd.1:Started 2024-09-13T05:17:59.768 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T05:17:59.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T05:17:59.770 INFO:tasks.ceph.osd.2:Started 2024-09-13T05:17:59.770 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T05:17:59.770 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T05:17:59.807 INFO:tasks.ceph.osd.3:Started 2024-09-13T05:17:59.807 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T05:17:59.808 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T05:17:59.810 INFO:tasks.ceph.osd.4:Started 2024-09-13T05:17:59.810 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T05:17:59.810 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T05:17:59.812 INFO:tasks.ceph.osd.5:Started 2024-09-13T05:17:59.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T05:17:59.997 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:17:59.997 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"fsid":"1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5","created":"2024-09-13T05:17:57.363809+0000","modified":"2024-09-13T05:17:59.738330+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T05:17:59.738330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e58174f4-85ec-4321-8bc3-384261c59c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84a3db6c-dacd-43d5-b964-37fdef0091a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0d530476-cf38-4544-9ce7-e8ed35df220b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fd91f7e2-580d-47ea-9db5-792c8f1ebe01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"01af0f4c-5da2-46fe-a262-59fcc6a2efa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"64cdd5f2-1568-4e96-ad90-6ef7414651f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T05:18:00.016 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T05:18:00.017 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T05:18:00.180 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:00.180+0000 7f4a8700b7c0 -1 Falling back to public interface 2024-09-13T05:18:00.186 INFO:tasks.ceph.mgr.x.smithi079.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-13T05:18:00.186 INFO:tasks.ceph.mgr.x.smithi079.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-13T05:18:00.186 INFO:tasks.ceph.mgr.x.smithi079.stderr: from numpy import show_config as show_numpy_config 2024-09-13T05:18:00.199 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:00.200+0000 7f16309f47c0 -1 Falling back to public interface 2024-09-13T05:18:00.221 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:00.220+0000 7f9baf137180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T05:18:00.356 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:00.356+0000 7f9baf137180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T05:18:00.374 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:00.371+0000 7f1341cc97c0 -1 Falling back to public interface 2024-09-13T05:18:00.378 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:00.375+0000 7fb3b476f7c0 -1 Falling back to public interface 2024-09-13T05:18:00.394 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:00.391+0000 7feb4643b7c0 -1 Falling back to public interface 2024-09-13T05:18:00.436 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:00.436+0000 7fa37f6007c0 -1 Falling back to public interface 2024-09-13T05:18:00.512 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:00.512+0000 7f9baf137180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T05:18:00.575 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:00.576+0000 7f9baf137180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T05:18:00.621 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T05:18:00.707 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:00.708+0000 7f9baf137180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T05:18:00.809 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-13T05:18:00.809 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":8,"fsid":"1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5","created":"2024-09-13T05:17:57.363809+0000","modified":"2024-09-13T05:17:59.738330+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T05:17:59.738330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e58174f4-85ec-4321-8bc3-384261c59c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84a3db6c-dacd-43d5-b964-37fdef0091a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0d530476-cf38-4544-9ce7-e8ed35df220b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fd91f7e2-580d-47ea-9db5-792c8f1ebe01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"01af0f4c-5da2-46fe-a262-59fcc6a2efa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"64cdd5f2-1568-4e96-ad90-6ef7414651f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T05:18:00.825 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-13T05:18:01.321 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:01.320+0000 7f9baf137180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T05:18:01.396 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:01.396+0000 7f9baf137180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T05:18:01.575 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:01.576+0000 7f9baf137180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T05:18:01.647 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:01.648+0000 7f9baf137180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T05:18:01.933 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:01.932+0000 7f9baf137180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T05:18:01.995 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:01.996+0000 7f9baf137180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T05:18:04.002 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:04.000+0000 7f16309f47c0 -1 osd.4 0 log_to_monitors true 2024-09-13T05:18:04.025 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:04.024+0000 7f4a8700b7c0 -1 osd.5 0 log_to_monitors true 2024-09-13T05:18:04.189 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:04.188+0000 7fa37f6007c0 -1 osd.3 0 log_to_monitors true 2024-09-13T05:18:04.201 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:04.195+0000 7f1341cc97c0 -1 osd.0 0 log_to_monitors true 2024-09-13T05:18:04.220 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:04.215+0000 7fb3b476f7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T05:18:04.305 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:04.303+0000 7feb4643b7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T05:18:05.087 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:05.088+0000 7f9b566ea640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T05:18:05.090 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:18:05.088+0000 7f9b566ea640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T05:18:06.766 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:06.764+0000 7f162c176640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T05:18:06.779 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:06.780+0000 7f4a8278d640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T05:18:06.801 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:06.800+0000 7f16247bb640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T05:18:06.807 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:06.808+0000 7f4a7add2640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T05:18:07.194 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:07.192+0000 7fa37ad82640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T05:18:07.227 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:07.228+0000 7fa3733c7640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T05:18:07.431 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T05:18:07.455 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:07.451+0000 7f133d44b640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T05:18:07.474 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:07.471+0000 7fb3afef1640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T05:18:07.506 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:07.503+0000 7fb3a8536640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T05:18:07.508 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:07.503+0000 7feb433f8640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T05:18:07.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:07.507+0000 7f1335a90640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T05:18:07.546 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:07.543+0000 7feb3a202640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T05:18:07.721 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-13T05:18:07.722 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":11,"fsid":"1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5","created":"2024-09-13T05:17:57.363809+0000","modified":"2024-09-13T05:18:06.060757+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T05:17:59.738330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e58174f4-85ec-4321-8bc3-384261c59c6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84a3db6c-dacd-43d5-b964-37fdef0091a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0d530476-cf38-4544-9ce7-e8ed35df220b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fd91f7e2-580d-47ea-9db5-792c8f1ebe01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"01af0f4c-5da2-46fe-a262-59fcc6a2efa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"64cdd5f2-1568-4e96-ad90-6ef7414651f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T05:18:07.737 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-13T05:18:14.344 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T05:18:14.643 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-13T05:18:14.643 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":15,"fsid":"1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5","created":"2024-09-13T05:17:57.363809+0000","modified":"2024-09-13T05:18:10.890960+0000","last_up_change":"2024-09-13T05:18:07.783213+0000","last_in_change":"2024-09-13T05:17:59.738330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T05:18:08.122164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e58174f4-85ec-4321-8bc3-384261c59c6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6801","nonce":2153409514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6803","nonce":2153409514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6807","nonce":2153409514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6805","nonce":2153409514}]},"public_addr":"172.21.15.16:6801/2153409514","cluster_addr":"172.21.15.16:6803/2153409514","heartbeat_back_addr":"172.21.15.16:6807/2153409514","heartbeat_front_addr":"172.21.15.16:6805/2153409514","state":["exists","up"]},{"osd":1,"uuid":"84a3db6c-dacd-43d5-b964-37fdef0091a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6809","nonce":2608485151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6811","nonce":2608485151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6815","nonce":2608485151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6813","nonce":2608485151}]},"public_addr":"172.21.15.16:6809/2608485151","cluster_addr":"172.21.15.16:6811/2608485151","heartbeat_back_addr":"172.21.15.16:6815/2608485151","heartbeat_front_addr":"172.21.15.16:6813/2608485151","state":["exists","up"]},{"osd":2,"uuid":"0d530476-cf38-4544-9ce7-e8ed35df220b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6817","nonce":1006404377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6819","nonce":1006404377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6823","nonce":1006404377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6821","nonce":1006404377}]},"public_addr":"172.21.15.16:6817/1006404377","cluster_addr":"172.21.15.16:6819/1006404377","heartbeat_back_addr":"172.21.15.16:6823/1006404377","heartbeat_front_addr":"172.21.15.16:6821/1006404377","state":["exists","up"]},{"osd":3,"uuid":"fd91f7e2-580d-47ea-9db5-792c8f1ebe01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6817","nonce":1731788850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6819","nonce":1731788850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6823","nonce":1731788850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6821","nonce":1731788850}]},"public_addr":"172.21.15.79:6817/1731788850","cluster_addr":"172.21.15.79:6819/1731788850","heartbeat_back_addr":"172.21.15.79:6823/1731788850","heartbeat_front_addr":"172.21.15.79:6821/1731788850","state":["exists","up"]},{"osd":4,"uuid":"01af0f4c-5da2-46fe-a262-59fcc6a2efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6809","nonce":2630134185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6811","nonce":2630134185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6815","nonce":2630134185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6813","nonce":2630134185}]},"public_addr":"172.21.15.79:6809/2630134185","cluster_addr":"172.21.15.79:6811/2630134185","heartbeat_back_addr":"172.21.15.79:6815/2630134185","heartbeat_front_addr":"172.21.15.79:6813/2630134185","state":["exists","up"]},{"osd":5,"uuid":"64cdd5f2-1568-4e96-ad90-6ef7414651f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6801","nonce":2778785304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6803","nonce":2778785304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6807","nonce":2778785304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6805","nonce":2778785304}]},"public_addr":"172.21.15.79:6801/2778785304","cluster_addr":"172.21.15.79:6803/2778785304","heartbeat_back_addr":"172.21.15.79:6807/2778785304","heartbeat_front_addr":"172.21.15.79:6805/2778785304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.193869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.255764+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.286354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.221310+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.026235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.010242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T05:18:14.659 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-13T05:18:14.660 INFO:tasks.ceph:Creating RBD pool 2024-09-13T05:18:14.660 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-13T05:18:15.113 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'rbd' created 2024-09-13T05:18:15.135 DEBUG:teuthology.orchestra.run.smithi016:> rbd --cluster ceph pool init rbd 2024-09-13T05:18:18.157 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T05:18:18.158 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T05:18:18.158 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T05:18:18.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T05:18:18.158 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T05:18:18.509 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:18.528 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1355891432},{"type":"v1","addr":"172.21.15.79:6825","nonce":1355891432}]},"active_addr":"172.21.15.79:6825/1355891432","active_change":"2024-09-13T05:18:04.037283+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":617303279}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":1032918293}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":17281544}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3689241973}]}]} 2024-09-13T05:18:18.530 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T05:18:18.530 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T05:18:18.530 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T05:18:18.829 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:18.830 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":19,"fsid":"1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5","created":"2024-09-13T05:17:57.363809+0000","modified":"2024-09-13T05:18:18.128520+0000","last_up_change":"2024-09-13T05:18:07.783213+0000","last_in_change":"2024-09-13T05:17:59.738330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T05:18:08.122164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T05:18:15.001664+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e58174f4-85ec-4321-8bc3-384261c59c6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6801","nonce":2153409514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6803","nonce":2153409514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6807","nonce":2153409514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6805","nonce":2153409514}]},"public_addr":"172.21.15.16:6801/2153409514","cluster_addr":"172.21.15.16:6803/2153409514","heartbeat_back_addr":"172.21.15.16:6807/2153409514","heartbeat_front_addr":"172.21.15.16:6805/2153409514","state":["exists","up"]},{"osd":1,"uuid":"84a3db6c-dacd-43d5-b964-37fdef0091a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6809","nonce":2608485151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6811","nonce":2608485151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6815","nonce":2608485151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6813","nonce":2608485151}]},"public_addr":"172.21.15.16:6809/2608485151","cluster_addr":"172.21.15.16:6811/2608485151","heartbeat_back_addr":"172.21.15.16:6815/2608485151","heartbeat_front_addr":"172.21.15.16:6813/2608485151","state":["exists","up"]},{"osd":2,"uuid":"0d530476-cf38-4544-9ce7-e8ed35df220b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6817","nonce":1006404377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6819","nonce":1006404377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6823","nonce":1006404377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6821","nonce":1006404377}]},"public_addr":"172.21.15.16:6817/1006404377","cluster_addr":"172.21.15.16:6819/1006404377","heartbeat_back_addr":"172.21.15.16:6823/1006404377","heartbeat_front_addr":"172.21.15.16:6821/1006404377","state":["exists","up"]},{"osd":3,"uuid":"fd91f7e2-580d-47ea-9db5-792c8f1ebe01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6817","nonce":1731788850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6819","nonce":1731788850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6823","nonce":1731788850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6821","nonce":1731788850}]},"public_addr":"172.21.15.79:6817/1731788850","cluster_addr":"172.21.15.79:6819/1731788850","heartbeat_back_addr":"172.21.15.79:6823/1731788850","heartbeat_front_addr":"172.21.15.79:6821/1731788850","state":["exists","up"]},{"osd":4,"uuid":"01af0f4c-5da2-46fe-a262-59fcc6a2efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6809","nonce":2630134185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6811","nonce":2630134185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6815","nonce":2630134185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6813","nonce":2630134185}]},"public_addr":"172.21.15.79:6809/2630134185","cluster_addr":"172.21.15.79:6811/2630134185","heartbeat_back_addr":"172.21.15.79:6815/2630134185","heartbeat_front_addr":"172.21.15.79:6813/2630134185","state":["exists","up"]},{"osd":5,"uuid":"64cdd5f2-1568-4e96-ad90-6ef7414651f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6801","nonce":2778785304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6803","nonce":2778785304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6807","nonce":2778785304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6805","nonce":2778785304}]},"public_addr":"172.21.15.79:6801/2778785304","cluster_addr":"172.21.15.79:6803/2778785304","heartbeat_back_addr":"172.21.15.79:6807/2778785304","heartbeat_front_addr":"172.21.15.79:6805/2778785304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.193869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.255764+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.286354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.221310+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.026235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.010242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T05:18:18.848 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T05:18:18.849 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T05:18:19.154 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:19.155 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":19,"fsid":"1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5","created":"2024-09-13T05:17:57.363809+0000","modified":"2024-09-13T05:18:18.128520+0000","last_up_change":"2024-09-13T05:18:07.783213+0000","last_in_change":"2024-09-13T05:17:59.738330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T05:18:08.122164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T05:18:15.001664+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e58174f4-85ec-4321-8bc3-384261c59c6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6801","nonce":2153409514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6803","nonce":2153409514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6807","nonce":2153409514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6805","nonce":2153409514}]},"public_addr":"172.21.15.16:6801/2153409514","cluster_addr":"172.21.15.16:6803/2153409514","heartbeat_back_addr":"172.21.15.16:6807/2153409514","heartbeat_front_addr":"172.21.15.16:6805/2153409514","state":["exists","up"]},{"osd":1,"uuid":"84a3db6c-dacd-43d5-b964-37fdef0091a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6809","nonce":2608485151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6811","nonce":2608485151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6815","nonce":2608485151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6813","nonce":2608485151}]},"public_addr":"172.21.15.16:6809/2608485151","cluster_addr":"172.21.15.16:6811/2608485151","heartbeat_back_addr":"172.21.15.16:6815/2608485151","heartbeat_front_addr":"172.21.15.16:6813/2608485151","state":["exists","up"]},{"osd":2,"uuid":"0d530476-cf38-4544-9ce7-e8ed35df220b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6817","nonce":1006404377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6819","nonce":1006404377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6823","nonce":1006404377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6821","nonce":1006404377}]},"public_addr":"172.21.15.16:6817/1006404377","cluster_addr":"172.21.15.16:6819/1006404377","heartbeat_back_addr":"172.21.15.16:6823/1006404377","heartbeat_front_addr":"172.21.15.16:6821/1006404377","state":["exists","up"]},{"osd":3,"uuid":"fd91f7e2-580d-47ea-9db5-792c8f1ebe01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6817","nonce":1731788850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6819","nonce":1731788850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6823","nonce":1731788850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1731788850},{"type":"v1","addr":"172.21.15.79:6821","nonce":1731788850}]},"public_addr":"172.21.15.79:6817/1731788850","cluster_addr":"172.21.15.79:6819/1731788850","heartbeat_back_addr":"172.21.15.79:6823/1731788850","heartbeat_front_addr":"172.21.15.79:6821/1731788850","state":["exists","up"]},{"osd":4,"uuid":"01af0f4c-5da2-46fe-a262-59fcc6a2efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6809","nonce":2630134185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6811","nonce":2630134185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6815","nonce":2630134185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":2630134185},{"type":"v1","addr":"172.21.15.79:6813","nonce":2630134185}]},"public_addr":"172.21.15.79:6809/2630134185","cluster_addr":"172.21.15.79:6811/2630134185","heartbeat_back_addr":"172.21.15.79:6815/2630134185","heartbeat_front_addr":"172.21.15.79:6813/2630134185","state":["exists","up"]},{"osd":5,"uuid":"64cdd5f2-1568-4e96-ad90-6ef7414651f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6801","nonce":2778785304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6803","nonce":2778785304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6807","nonce":2778785304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6805","nonce":2778785304}]},"public_addr":"172.21.15.79:6801/2778785304","cluster_addr":"172.21.15.79:6803/2778785304","heartbeat_back_addr":"172.21.15.79:6807/2778785304","heartbeat_front_addr":"172.21.15.79:6805/2778785304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.193869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.255764+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.286354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.221310+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.026235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.010242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T05:18:19.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T05:18:19.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T05:18:19.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T05:18:19.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T05:18:19.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T05:18:19.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T05:18:19.361 INFO:teuthology.orchestra.run.smithi016.stdout:51539607558 2024-09-13T05:18:19.362 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T05:18:19.367 INFO:teuthology.orchestra.run.smithi016.stdout:51539607559 2024-09-13T05:18:19.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T05:18:19.391 INFO:teuthology.orchestra.run.smithi016.stdout:51539607558 2024-09-13T05:18:19.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T05:18:19.423 INFO:teuthology.orchestra.run.smithi016.stdout:51539607559 2024-09-13T05:18:19.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T05:18:19.433 INFO:teuthology.orchestra.run.smithi016.stdout:51539607558 2024-09-13T05:18:19.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T05:18:19.437 INFO:teuthology.orchestra.run.smithi016.stdout:51539607558 2024-09-13T05:18:19.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T05:18:19.771 INFO:teuthology.orchestra.run.smithi016.stdout:51539607556 2024-09-13T05:18:19.788 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.3 2024-09-13T05:18:19.807 INFO:teuthology.orchestra.run.smithi016.stdout:51539607557 2024-09-13T05:18:19.818 INFO:teuthology.orchestra.run.smithi016.stdout:51539607557 2024-09-13T05:18:19.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607559 got 51539607557 for osd.4 2024-09-13T05:18:19.836 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607559 got 51539607557 for osd.5 2024-09-13T05:18:19.848 INFO:teuthology.orchestra.run.smithi016.stdout:51539607556 2024-09-13T05:18:19.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.2 2024-09-13T05:18:19.897 INFO:teuthology.orchestra.run.smithi016.stdout:51539607556 2024-09-13T05:18:19.911 INFO:teuthology.orchestra.run.smithi016.stdout:51539607556 2024-09-13T05:18:19.913 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.1 2024-09-13T05:18:19.927 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.0 2024-09-13T05:18:20.789 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T05:18:20.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T05:18:20.837 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T05:18:20.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T05:18:20.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T05:18:20.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T05:18:21.194 INFO:teuthology.orchestra.run.smithi016.stdout:51539607559 2024-09-13T05:18:21.195 INFO:teuthology.orchestra.run.smithi016.stdout:51539607558 2024-09-13T05:18:21.216 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607559 got 51539607559 for osd.4 2024-09-13T05:18:21.216 DEBUG:teuthology.parallel:result is None 2024-09-13T05:18:21.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.2 2024-09-13T05:18:21.217 DEBUG:teuthology.parallel:result is None 2024-09-13T05:18:21.249 INFO:teuthology.orchestra.run.smithi016.stdout:51539607558 2024-09-13T05:18:21.261 INFO:teuthology.orchestra.run.smithi016.stdout:51539607559 2024-09-13T05:18:21.268 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.3 2024-09-13T05:18:21.268 DEBUG:teuthology.parallel:result is None 2024-09-13T05:18:21.279 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607559 got 51539607559 for osd.5 2024-09-13T05:18:21.279 DEBUG:teuthology.parallel:result is None 2024-09-13T05:18:21.321 INFO:teuthology.orchestra.run.smithi016.stdout:51539607558 2024-09-13T05:18:21.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.0 2024-09-13T05:18:21.338 DEBUG:teuthology.parallel:result is None 2024-09-13T05:18:21.339 INFO:teuthology.orchestra.run.smithi016.stdout:51539607558 2024-09-13T05:18:21.355 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.1 2024-09-13T05:18:21.356 DEBUG:teuthology.parallel:result is None 2024-09-13T05:18:21.356 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T05:18:21.356 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:18:21.674 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:21.674 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:18:21.691 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T05:18:20.050866+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176652,"kb_used_data":2224,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054388,"statfs":{"total":579820584960,"available":579639693312,"internally_reserved":0,"allocated":2277376,"data_stored":1536238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.937677"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324112+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:18:18.324112+0000","last_peered":"2024-09-13T05:18:18.324112+0000","last_clean":"2024-09-13T05:18:18.324112+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:18:18.324112+0000","last_undegraded":"2024-09-13T05:18:18.324112+0000","last_fullsized":"2024-09-13T05:18:18.324112+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.142953+0000","last_change":"2024-09-13T05:18:18.143477+0000","last_active":"2024-09-13T05:18:18.142953+0000","last_peered":"2024-09-13T05:18:18.142953+0000","last_clean":"2024-09-13T05:18:18.142953+0000","last_became_active":"2024-09-13T05:18:16.129462+0000","last_became_peered":"2024-09-13T05:18:16.129462+0000","last_unstale":"2024-09-13T05:18:18.142953+0000","last_undegraded":"2024-09-13T05:18:18.142953+0000","last_fullsized":"2024-09-13T05:18:18.142953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:33:02.115421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076334099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324086+0000","last_change":"2024-09-13T05:18:18.324354+0000","last_active":"2024-09-13T05:18:18.324086+0000","last_peered":"2024-09-13T05:18:18.324086+0000","last_clean":"2024-09-13T05:18:18.324086+0000","last_became_active":"2024-09-13T05:18:16.125619+0000","last_became_peered":"2024-09-13T05:18:16.125619+0000","last_unstale":"2024-09-13T05:18:18.324086+0000","last_undegraded":"2024-09-13T05:18:18.324086+0000","last_fullsized":"2024-09-13T05:18:18.324086+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:40.629120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.293120+0000","last_change":"2024-09-13T05:18:18.293687+0000","last_active":"2024-09-13T05:18:18.293120+0000","last_peered":"2024-09-13T05:18:18.293120+0000","last_clean":"2024-09-13T05:18:18.293120+0000","last_became_active":"2024-09-13T05:18:16.130068+0000","last_became_peered":"2024-09-13T05:18:16.130068+0000","last_unstale":"2024-09-13T05:18:18.293120+0000","last_undegraded":"2024-09-13T05:18:18.293120+0000","last_fullsized":"2024-09-13T05:18:18.293120+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:44:56.391514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.147989+0000","last_change":"2024-09-13T05:18:18.147989+0000","last_active":"2024-09-13T05:18:18.147989+0000","last_peered":"2024-09-13T05:18:18.147989+0000","last_clean":"2024-09-13T05:18:18.147989+0000","last_became_active":"2024-09-13T05:18:16.129277+0000","last_became_peered":"2024-09-13T05:18:16.129277+0000","last_unstale":"2024-09-13T05:18:18.147989+0000","last_undegraded":"2024-09-13T05:18:18.147989+0000","last_fullsized":"2024-09-13T05:18:18.147989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:59:39.922642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.734711+0000","last_change":"2024-09-13T05:18:18.735084+0000","last_active":"2024-09-13T05:18:18.734711+0000","last_peered":"2024-09-13T05:18:18.734711+0000","last_clean":"2024-09-13T05:18:18.734711+0000","last_became_active":"2024-09-13T05:18:16.129951+0000","last_became_peered":"2024-09-13T05:18:16.129951+0000","last_unstale":"2024-09-13T05:18:18.734711+0000","last_undegraded":"2024-09-13T05:18:18.734711+0000","last_fullsized":"2024-09-13T05:18:18.734711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:14:50.648935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063472099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.140525+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:18:18.140525+0000","last_peered":"2024-09-13T05:18:18.140525+0000","last_clean":"2024-09-13T05:18:18.140525+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:18:18.140525+0000","last_undegraded":"2024-09-13T05:18:18.140525+0000","last_fullsized":"2024-09-13T05:18:18.140525+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324564+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:18:18.324564+0000","last_peered":"2024-09-13T05:18:18.324564+0000","last_clean":"2024-09-13T05:18:18.324564+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:18:18.324564+0000","last_undegraded":"2024-09-13T05:18:18.324564+0000","last_fullsized":"2024-09-13T05:18:18.324564+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T05:18:17.310780+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:18:17.310780+0000","last_peered":"2024-09-13T05:18:17.310780+0000","last_clean":"2024-09-13T05:18:17.310780+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:18:17.310780+0000","last_undegraded":"2024-09-13T05:18:17.310780+0000","last_fullsized":"2024-09-13T05:18:17.310780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607559,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":649612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":59263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":180224,"data_stored":59244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":774144,"data_stored":649612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]}]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":184320,"data_stored":59263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:18:21.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:18:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:21.977 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:18:21.993 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T05:18:20.050866+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176652,"kb_used_data":2224,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054388,"statfs":{"total":579820584960,"available":579639693312,"internally_reserved":0,"allocated":2277376,"data_stored":1536238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.937677"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324112+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:18:18.324112+0000","last_peered":"2024-09-13T05:18:18.324112+0000","last_clean":"2024-09-13T05:18:18.324112+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:18:18.324112+0000","last_undegraded":"2024-09-13T05:18:18.324112+0000","last_fullsized":"2024-09-13T05:18:18.324112+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.142953+0000","last_change":"2024-09-13T05:18:18.143477+0000","last_active":"2024-09-13T05:18:18.142953+0000","last_peered":"2024-09-13T05:18:18.142953+0000","last_clean":"2024-09-13T05:18:18.142953+0000","last_became_active":"2024-09-13T05:18:16.129462+0000","last_became_peered":"2024-09-13T05:18:16.129462+0000","last_unstale":"2024-09-13T05:18:18.142953+0000","last_undegraded":"2024-09-13T05:18:18.142953+0000","last_fullsized":"2024-09-13T05:18:18.142953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:33:02.115421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076334099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324086+0000","last_change":"2024-09-13T05:18:18.324354+0000","last_active":"2024-09-13T05:18:18.324086+0000","last_peered":"2024-09-13T05:18:18.324086+0000","last_clean":"2024-09-13T05:18:18.324086+0000","last_became_active":"2024-09-13T05:18:16.125619+0000","last_became_peered":"2024-09-13T05:18:16.125619+0000","last_unstale":"2024-09-13T05:18:18.324086+0000","last_undegraded":"2024-09-13T05:18:18.324086+0000","last_fullsized":"2024-09-13T05:18:18.324086+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:40.629120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.293120+0000","last_change":"2024-09-13T05:18:18.293687+0000","last_active":"2024-09-13T05:18:18.293120+0000","last_peered":"2024-09-13T05:18:18.293120+0000","last_clean":"2024-09-13T05:18:18.293120+0000","last_became_active":"2024-09-13T05:18:16.130068+0000","last_became_peered":"2024-09-13T05:18:16.130068+0000","last_unstale":"2024-09-13T05:18:18.293120+0000","last_undegraded":"2024-09-13T05:18:18.293120+0000","last_fullsized":"2024-09-13T05:18:18.293120+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:44:56.391514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.147989+0000","last_change":"2024-09-13T05:18:18.147989+0000","last_active":"2024-09-13T05:18:18.147989+0000","last_peered":"2024-09-13T05:18:18.147989+0000","last_clean":"2024-09-13T05:18:18.147989+0000","last_became_active":"2024-09-13T05:18:16.129277+0000","last_became_peered":"2024-09-13T05:18:16.129277+0000","last_unstale":"2024-09-13T05:18:18.147989+0000","last_undegraded":"2024-09-13T05:18:18.147989+0000","last_fullsized":"2024-09-13T05:18:18.147989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:59:39.922642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.734711+0000","last_change":"2024-09-13T05:18:18.735084+0000","last_active":"2024-09-13T05:18:18.734711+0000","last_peered":"2024-09-13T05:18:18.734711+0000","last_clean":"2024-09-13T05:18:18.734711+0000","last_became_active":"2024-09-13T05:18:16.129951+0000","last_became_peered":"2024-09-13T05:18:16.129951+0000","last_unstale":"2024-09-13T05:18:18.734711+0000","last_undegraded":"2024-09-13T05:18:18.734711+0000","last_fullsized":"2024-09-13T05:18:18.734711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:14:50.648935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063472099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.140525+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:18:18.140525+0000","last_peered":"2024-09-13T05:18:18.140525+0000","last_clean":"2024-09-13T05:18:18.140525+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:18:18.140525+0000","last_undegraded":"2024-09-13T05:18:18.140525+0000","last_fullsized":"2024-09-13T05:18:18.140525+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324564+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:18:18.324564+0000","last_peered":"2024-09-13T05:18:18.324564+0000","last_clean":"2024-09-13T05:18:18.324564+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:18:18.324564+0000","last_undegraded":"2024-09-13T05:18:18.324564+0000","last_fullsized":"2024-09-13T05:18:18.324564+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T05:18:17.310780+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:18:17.310780+0000","last_peered":"2024-09-13T05:18:17.310780+0000","last_clean":"2024-09-13T05:18:17.310780+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:18:17.310780+0000","last_undegraded":"2024-09-13T05:18:17.310780+0000","last_fullsized":"2024-09-13T05:18:17.310780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607559,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":649612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":59263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":180224,"data_stored":59244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":774144,"data_stored":649612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]}]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":184320,"data_stored":59263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:18:21.994 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T05:18:21.995 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T05:18:21.995 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T05:18:21.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T05:18:22.311 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:22.311 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T05:18:22.329 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T05:18:22.329 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-13T05:18:22.339 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T05:18:22.339 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '7a853b1706d076d58e0da7a5d3b9b9eb36276880'}} 2024-09-13T05:18:22.340 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-13T05:18:22.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-13T05:18:22.649 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T05:18:22.649 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T05:18:22.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:18:22.942 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:22.943 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:18:22.960 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T05:18:22.051461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176652,"kb_used_data":2224,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054388,"statfs":{"total":579820584960,"available":579639693312,"internally_reserved":0,"allocated":2277376,"data_stored":1536238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.935901"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324112+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:18:18.324112+0000","last_peered":"2024-09-13T05:18:18.324112+0000","last_clean":"2024-09-13T05:18:18.324112+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:18:18.324112+0000","last_undegraded":"2024-09-13T05:18:18.324112+0000","last_fullsized":"2024-09-13T05:18:18.324112+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.142953+0000","last_change":"2024-09-13T05:18:18.143477+0000","last_active":"2024-09-13T05:18:18.142953+0000","last_peered":"2024-09-13T05:18:18.142953+0000","last_clean":"2024-09-13T05:18:18.142953+0000","last_became_active":"2024-09-13T05:18:16.129462+0000","last_became_peered":"2024-09-13T05:18:16.129462+0000","last_unstale":"2024-09-13T05:18:18.142953+0000","last_undegraded":"2024-09-13T05:18:18.142953+0000","last_fullsized":"2024-09-13T05:18:18.142953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:33:02.115421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076334099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324086+0000","last_change":"2024-09-13T05:18:18.324354+0000","last_active":"2024-09-13T05:18:18.324086+0000","last_peered":"2024-09-13T05:18:18.324086+0000","last_clean":"2024-09-13T05:18:18.324086+0000","last_became_active":"2024-09-13T05:18:16.125619+0000","last_became_peered":"2024-09-13T05:18:16.125619+0000","last_unstale":"2024-09-13T05:18:18.324086+0000","last_undegraded":"2024-09-13T05:18:18.324086+0000","last_fullsized":"2024-09-13T05:18:18.324086+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:40.629120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.293120+0000","last_change":"2024-09-13T05:18:18.293687+0000","last_active":"2024-09-13T05:18:18.293120+0000","last_peered":"2024-09-13T05:18:18.293120+0000","last_clean":"2024-09-13T05:18:18.293120+0000","last_became_active":"2024-09-13T05:18:16.130068+0000","last_became_peered":"2024-09-13T05:18:16.130068+0000","last_unstale":"2024-09-13T05:18:18.293120+0000","last_undegraded":"2024-09-13T05:18:18.293120+0000","last_fullsized":"2024-09-13T05:18:18.293120+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:44:56.391514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.147989+0000","last_change":"2024-09-13T05:18:18.147989+0000","last_active":"2024-09-13T05:18:18.147989+0000","last_peered":"2024-09-13T05:18:18.147989+0000","last_clean":"2024-09-13T05:18:18.147989+0000","last_became_active":"2024-09-13T05:18:16.129277+0000","last_became_peered":"2024-09-13T05:18:16.129277+0000","last_unstale":"2024-09-13T05:18:18.147989+0000","last_undegraded":"2024-09-13T05:18:18.147989+0000","last_fullsized":"2024-09-13T05:18:18.147989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:59:39.922642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.734711+0000","last_change":"2024-09-13T05:18:18.735084+0000","last_active":"2024-09-13T05:18:18.734711+0000","last_peered":"2024-09-13T05:18:18.734711+0000","last_clean":"2024-09-13T05:18:18.734711+0000","last_became_active":"2024-09-13T05:18:16.129951+0000","last_became_peered":"2024-09-13T05:18:16.129951+0000","last_unstale":"2024-09-13T05:18:18.734711+0000","last_undegraded":"2024-09-13T05:18:18.734711+0000","last_fullsized":"2024-09-13T05:18:18.734711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:14:50.648935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063472099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.140525+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:18:18.140525+0000","last_peered":"2024-09-13T05:18:18.140525+0000","last_clean":"2024-09-13T05:18:18.140525+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:18:18.140525+0000","last_undegraded":"2024-09-13T05:18:18.140525+0000","last_fullsized":"2024-09-13T05:18:18.140525+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324564+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:18:18.324564+0000","last_peered":"2024-09-13T05:18:18.324564+0000","last_clean":"2024-09-13T05:18:18.324564+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:18:18.324564+0000","last_undegraded":"2024-09-13T05:18:18.324564+0000","last_fullsized":"2024-09-13T05:18:18.324564+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T05:18:17.310780+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:18:17.310780+0000","last_peered":"2024-09-13T05:18:17.310780+0000","last_clean":"2024-09-13T05:18:17.310780+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:18:17.310780+0000","last_undegraded":"2024-09-13T05:18:17.310780+0000","last_fullsized":"2024-09-13T05:18:17.310780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607559,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":649612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":59263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":180224,"data_stored":59244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":774144,"data_stored":649612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]}]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":184320,"data_stored":59263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:18:22.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:18:23.253 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:18:23.270 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T05:18:22.051461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":176652,"kb_used_data":2224,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":566054388,"statfs":{"total":579820584960,"available":579639693312,"internally_reserved":0,"allocated":2277376,"data_stored":1536238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.935901"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324112+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:18:18.324112+0000","last_peered":"2024-09-13T05:18:18.324112+0000","last_clean":"2024-09-13T05:18:18.324112+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:18:18.324112+0000","last_undegraded":"2024-09-13T05:18:18.324112+0000","last_fullsized":"2024-09-13T05:18:18.324112+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.142953+0000","last_change":"2024-09-13T05:18:18.143477+0000","last_active":"2024-09-13T05:18:18.142953+0000","last_peered":"2024-09-13T05:18:18.142953+0000","last_clean":"2024-09-13T05:18:18.142953+0000","last_became_active":"2024-09-13T05:18:16.129462+0000","last_became_peered":"2024-09-13T05:18:16.129462+0000","last_unstale":"2024-09-13T05:18:18.142953+0000","last_undegraded":"2024-09-13T05:18:18.142953+0000","last_fullsized":"2024-09-13T05:18:18.142953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:33:02.115421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076334099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324086+0000","last_change":"2024-09-13T05:18:18.324354+0000","last_active":"2024-09-13T05:18:18.324086+0000","last_peered":"2024-09-13T05:18:18.324086+0000","last_clean":"2024-09-13T05:18:18.324086+0000","last_became_active":"2024-09-13T05:18:16.125619+0000","last_became_peered":"2024-09-13T05:18:16.125619+0000","last_unstale":"2024-09-13T05:18:18.324086+0000","last_undegraded":"2024-09-13T05:18:18.324086+0000","last_fullsized":"2024-09-13T05:18:18.324086+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:40.629120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.293120+0000","last_change":"2024-09-13T05:18:18.293687+0000","last_active":"2024-09-13T05:18:18.293120+0000","last_peered":"2024-09-13T05:18:18.293120+0000","last_clean":"2024-09-13T05:18:18.293120+0000","last_became_active":"2024-09-13T05:18:16.130068+0000","last_became_peered":"2024-09-13T05:18:16.130068+0000","last_unstale":"2024-09-13T05:18:18.293120+0000","last_undegraded":"2024-09-13T05:18:18.293120+0000","last_fullsized":"2024-09-13T05:18:18.293120+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:44:56.391514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.147989+0000","last_change":"2024-09-13T05:18:18.147989+0000","last_active":"2024-09-13T05:18:18.147989+0000","last_peered":"2024-09-13T05:18:18.147989+0000","last_clean":"2024-09-13T05:18:18.147989+0000","last_became_active":"2024-09-13T05:18:16.129277+0000","last_became_peered":"2024-09-13T05:18:16.129277+0000","last_unstale":"2024-09-13T05:18:18.147989+0000","last_undegraded":"2024-09-13T05:18:18.147989+0000","last_fullsized":"2024-09-13T05:18:18.147989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:59:39.922642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.734711+0000","last_change":"2024-09-13T05:18:18.735084+0000","last_active":"2024-09-13T05:18:18.734711+0000","last_peered":"2024-09-13T05:18:18.734711+0000","last_clean":"2024-09-13T05:18:18.734711+0000","last_became_active":"2024-09-13T05:18:16.129951+0000","last_became_peered":"2024-09-13T05:18:16.129951+0000","last_unstale":"2024-09-13T05:18:18.734711+0000","last_undegraded":"2024-09-13T05:18:18.734711+0000","last_fullsized":"2024-09-13T05:18:18.734711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:14:50.648935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063472099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.140525+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:18:18.140525+0000","last_peered":"2024-09-13T05:18:18.140525+0000","last_clean":"2024-09-13T05:18:18.140525+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:18:18.140525+0000","last_undegraded":"2024-09-13T05:18:18.140525+0000","last_fullsized":"2024-09-13T05:18:18.140525+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T05:18:18.324564+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:18:18.324564+0000","last_peered":"2024-09-13T05:18:18.324564+0000","last_clean":"2024-09-13T05:18:18.324564+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:18:18.324564+0000","last_undegraded":"2024-09-13T05:18:18.324564+0000","last_fullsized":"2024-09-13T05:18:18.324564+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T05:18:17.310780+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:18:17.310780+0000","last_peered":"2024-09-13T05:18:17.310780+0000","last_clean":"2024-09-13T05:18:17.310780+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:18:17.310780+0000","last_undegraded":"2024-09-13T05:18:17.310780+0000","last_fullsized":"2024-09-13T05:18:17.310780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607559,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":649612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":59263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":180224,"data_stored":59244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":774144,"data_stored":649612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]}]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":184320,"data_stored":59263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:18:23.271 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T05:18:23.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-13T05:18:23.565 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 20 2024-09-13T05:18:23.565 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5 2024-09-13T05:18:23.565 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-13T05:17:57.363809+0000 2024-09-13T05:18:23.565 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-09-13T05:18:22.006991+0000 2024-09-13T05:18:23.565 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 5 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client jewel 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release reef 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 15 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.06 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 19 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.25 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 6 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/2153409514,v1:172.21.15.16:6801/2153409514] [v2:172.21.15.16:6802/2153409514,v1:172.21.15.16:6803/2153409514] exists,up e58174f4-85ec-4321-8bc3-384261c59c6b 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/2608485151,v1:172.21.15.16:6809/2608485151] [v2:172.21.15.16:6810/2608485151,v1:172.21.15.16:6811/2608485151] exists,up 84a3db6c-dacd-43d5-b964-37fdef0091a4 2024-09-13T05:18:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6816/1006404377,v1:172.21.15.16:6817/1006404377] [v2:172.21.15.16:6818/1006404377,v1:172.21.15.16:6819/1006404377] exists,up 0d530476-cf38-4544-9ce7-e8ed35df220b 2024-09-13T05:18:23.567 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6816/1731788850,v1:172.21.15.79:6817/1731788850] [v2:172.21.15.79:6818/1731788850,v1:172.21.15.79:6819/1731788850] exists,up fd91f7e2-580d-47ea-9db5-792c8f1ebe01 2024-09-13T05:18:23.567 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6808/2630134185,v1:172.21.15.79:6809/2630134185] [v2:172.21.15.79:6810/2630134185,v1:172.21.15.79:6811/2630134185] exists,up 01af0f4c-5da2-46fe-a262-59fcc6a2efa9 2024-09-13T05:18:23.567 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6800/2778785304,v1:172.21.15.79:6801/2778785304] [v2:172.21.15.79:6802/2778785304,v1:172.21.15.79:6803/2778785304] exists,up 64cdd5f2-1568-4e96-ad90-6ef7414651f0 2024-09-13T05:18:23.583 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/2153409514,v1:172.21.15.16:6801/2153409514] [v2:172.21.15.16:6802/2153409514,v1:172.21.15.16:6803/2153409514] exists,up e58174f4-85ec-4321-8bc3-384261c59c6b', 'osd.1 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/2608485151,v1:172.21.15.16:6809/2608485151] [v2:172.21.15.16:6810/2608485151,v1:172.21.15.16:6811/2608485151] exists,up 84a3db6c-dacd-43d5-b964-37fdef0091a4', 'osd.2 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6816/1006404377,v1:172.21.15.16:6817/1006404377] [v2:172.21.15.16:6818/1006404377,v1:172.21.15.16:6819/1006404377] exists,up 0d530476-cf38-4544-9ce7-e8ed35df220b', 'osd.3 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6816/1731788850,v1:172.21.15.79:6817/1731788850] [v2:172.21.15.79:6818/1731788850,v1:172.21.15.79:6819/1731788850] exists,up fd91f7e2-580d-47ea-9db5-792c8f1ebe01', 'osd.4 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6808/2630134185,v1:172.21.15.79:6809/2630134185] [v2:172.21.15.79:6810/2630134185,v1:172.21.15.79:6811/2630134185] exists,up 01af0f4c-5da2-46fe-a262-59fcc6a2efa9', 'osd.5 up in weight 1 up_from 12 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6800/2778785304,v1:172.21.15.79:6801/2778785304] [v2:172.21.15.79:6802/2778785304,v1:172.21.15.79:6803/2778785304] exists,up 64cdd5f2-1568-4e96-ad90-6ef7414651f0'] 2024-09-13T05:18:23.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-13T05:18:23.704 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:23.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mon.a", 2024-09-13T05:18:23.704 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster": "ceph", 2024-09-13T05:18:23.704 INFO:teuthology.orchestra.run.smithi016.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T05:18:23.704 INFO:teuthology.orchestra.run.smithi016.stdout: "admin_socket_mode": "", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_client_required": "cephx, none", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_cluster_required": "cephx", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_debug": "true", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_service_required": "cephx", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_supported": "cephx", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio": "true", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T05:18:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio_reap_max": "16", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_async_discard": "false", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_block_size": "4096", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_aio": "false", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_enable_discard": "false", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_flock_retry": "3", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_inject_crash": "0", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_ioring": "false", 2024-09-13T05:18:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_ioring_hipri": "false", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_type": "", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_allocator": "hybrid", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_buffered_io": "true", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T05:18:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_replay_recovery": "false", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_sync_write": "false", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T05:18:23.708 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_allocator": "hybrid", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_create": "true", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_db_create": "false", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_db_path": "", 2024-09-13T05:18:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_db_size": "0", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_path": "", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_size": "107374182400", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_wal_create": "false", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_wal_path": "", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs": "true", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T05:18:23.710 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_autotune": "true", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_size": "0", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T05:18:23.711 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_type": "2q", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_clone_cow": "true", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_mode": "none", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T05:18:23.712 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_freelist": "false", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-13T05:18:23.713 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T05:18:23.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fail_eio": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T05:18:23.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_blob_size": "0", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T05:18:23.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T05:18:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_mem": "512", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T05:18:23.718 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_tracing": "false", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T05:18:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T05:18:23.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_cluster_require_version": "2", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_require_signatures": "false", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_require_version": "2", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_service_require_signatures": "false", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_service_require_version": "2", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_sign_messages": "true", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "chdir": "", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "client_acl_type": "", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "client_asio_thread_count": "2", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "client_cache_mid": "0.750000", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "client_cache_size": "16384", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "client_caps_release_delay": "5", 2024-09-13T05:18:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: "client_check_pool_perm": "true", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_force_sync_read": "false", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_getattr_caps": "false", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_inject_features": "", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_die_on_failed_remount": "false", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_dirsize_rbytes": "true", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_force_lazyio": "false", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_fs": "", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_inject_release_failure": "false", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_max_inline_size": "4096", 2024-09-13T05:18:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mds_namespace": "", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_metadata": "", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mount_gid": "-1", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mount_timeout": "300", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mount_uid": "-1", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mountpoint": "/", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_notify_timeout": "10", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc": "true", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_max_objects": "1000", 2024-09-13T05:18:23.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_size": "209715200", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_permissions": "true", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_quota": "true", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_quota_df": "true", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_readahead_max_bytes": "0", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_readahead_max_periods": "4", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_readahead_min": "131072", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_reconnect_stale": "false", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_shutdown_timeout": "30", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_snapdir": ".snap", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_tick_interval": "1", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_trace": "", 2024-09-13T05:18:23.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_use_faked_inos": "false", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_use_random_mds": "false", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_graylog": "false", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_graylog_port": "12201", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_monitors": "default=true", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_syslog": "false", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_syslog_level": "info", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "-", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_network": "", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_network_interface": "", 2024-09-13T05:18:23.725 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zlib_isal": "false", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zlib_level": "5", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zstd_level": "1", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_seastar_smp": "1", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_location": "", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_location_hook": "", 2024-09-13T05:18:23.726 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_location_hook_timeout": "10", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "daemonize": "false", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "daos_pool": "tank", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_alienstore": "0/5", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_asok": "1/5", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_asok_assert_abort": "false", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_auth": "20/20", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_bdev": "1/3", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_bluefs": "1/5", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_bluestore": "1/5", 2024-09-13T05:18:23.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_buffer": "0/1", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_cephsqlite": "0/5", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_client": "0/5", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_compressor": "1/5", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_context": "0/1", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_crush": "1/1", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_crypto": "1/5", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_cyanstore": "0/5", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_dpdk": "1/5", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_eventtrace": "1/5", 2024-09-13T05:18:23.728 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_filer": "0/1", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_filestore": "1/3", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_finisher": "1/1", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_fuse": "1/5", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_javaclient": "1/5", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_journal": "1/3", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_journaler": "0/5", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_kstore": "1/5", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_leveldb": "4/5", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_lockdep": "0/1", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mclock": "1/5", 2024-09-13T05:18:23.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds": "1/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_balancer": "1/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_locker": "1/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_log": "1/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_migrator": "1/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_memstore": "1/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mgr": "2/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mgrc": "1/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mon": "20/20", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_monc": "0/10", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_ms": "1/1", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_none": "0/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_objclass": "0/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_objectcacher": "0/5", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_objecter": "0/1", 2024-09-13T05:18:23.730 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_optracker": "0/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_osd": "1/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_paxos": "20/20", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_perfcounter": "1/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_prioritycache": "1/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rados": "0/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd": "0/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd_replay": "0/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_refs": "0/0", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_reserver": "1/1", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw": "1/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_access": "1/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T05:18:23.731 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_flight": "1/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_sync": "1/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rocksdb": "4/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_backref": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_cache": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_device": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_epm": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_journal": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_lba": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_odata": "0/5", 2024-09-13T05:18:23.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_omap": "0/5", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_onode": "0/5", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_t": "0/5", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_tm": "0/5", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_striper": "0/1", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_test": "0/5", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_throttle": "1/1", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_timer": "0/1", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_tp": "0/5", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "device_failure_prediction_mode": "none", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-13T05:18:23.733 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_graylog": "false", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_journald": "false", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_stderr": "true", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_syslog": "false", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event_tracing": "false", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_http_port": "9926", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_prio_limit": "5", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_sort_metrics": "true", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_stats_period": "5", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "fake_statfs_for_testing": "0", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "fatal_signal_handlers": "true", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "filer_max_purge_ops": "10", 2024-09-13T05:18:23.734 INFO:teuthology.orchestra.run.smithi016.stdout: "filer_max_truncate_ops": "128", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_blackhole": "false", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_btrfs_snap": "true", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_caller_concurrency": "10", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_omap_check": "false", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_verify_split": "false", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_dump_file": "", 2024-09-13T05:18:23.735 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fadvise": "true", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fail_eio": "true", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fd_cache_size": "128", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fiemap": "false", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_inject_stall": "0", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_journal_parallel": "false", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_journal_trailing": "false", 2024-09-13T05:18:23.736 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_journal_writeahead": "false", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_kill_at": "0", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T05:18:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_merge_threshold": "-10", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_odsync_write": "false", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_omap_backend_path": "", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_op_threads": "2", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_punch_hole": "false", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T05:18:23.738 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_ops": "50", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_seek_data_hole": "false", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_sloppy_crc": "false", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T05:18:23.739 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_splice": "false", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_split_multiple": "2", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_split_rand_factor": "20", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_update_to": "1000", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T05:18:23.740 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_xfs_extsize": "false", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_zfs_snap": "false", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "fio_dir": "/tmp/fio", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_allow_other": "true", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_big_writes": "true", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_debug": "false", 2024-09-13T05:18:23.741 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_default_permissions": "false", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_disable_pagecache": "false", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_max_write": "0", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_multithreaded": "true", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_require_active_mds": "true", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_set_user_groups": "true", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_splice_move": "true", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_splice_read": "true", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_splice_write": "true", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "gss_target_name": "ceph", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_file": "", 2024-09-13T05:18:23.742 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_inject_failure": "0", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_interval": "5", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "host": "smithi016", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T05:18:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "inject_early_sigterm": "false", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "jaeger_agent_port": "6799", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "jaeger_tracing_enable": "false", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_aio": "true", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_align_min_size": "65536", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_block_align": "true", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_block_size": "4096", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_dio": "true", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_discard": "false", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_force_aio": "false", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_ignore_corruption": "false", 2024-09-13T05:18:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_max_write_entries": "100", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_replay_from": "0", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_write_header_frequency": "0", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_zero_on_create": "false", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journaler_prefetch_periods": "10", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journaler_prezero_periods": "5", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "journaler_write_head_interval": "15", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "keyfile": "", 2024-09-13T05:18:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_backend": "rocksdb", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_max_bytes": "67108864", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_max_ops": "512", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_onode_map_size": "1024", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_sync_transaction": "false", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_block_size": "65536", 2024-09-13T05:18:23.746 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_bloom_size": "0", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_cache_size": "536870912", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_compact_on_mount": "false", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_compression": "false", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_log": "", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_max_open_files": "0", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_paranoid": "false", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "librados_thread_count": "2", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "lockdep": "false", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "lockdep_force_backtrace": "false", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "log_coarse_timestamps": "true", 2024-09-13T05:18:23.747 INFO:teuthology.orchestra.run.smithi016.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T05:18:23.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_flush_on_exit": "false", 2024-09-13T05:18:23.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T05:18:23.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_graylog_port": "12201", 2024-09-13T05:18:23.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_max_new": "1000", 2024-09-13T05:18:23.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_max_recent": "10000", 2024-09-13T05:18:23.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_stderr_prefix": "", 2024-09-13T05:18:23.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T05:18:23.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": "true", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_graylog": "false", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_journald": "false", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_stderr": "false", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_syslog": "false", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_action_on_write_error": "1", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_alternate_name_max": "8192", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_asio_thread_count": "2", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_export_pin": "true", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T05:18:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_interval": "10", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_max": "-1", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_max_until": "-1", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_merge_size": "50", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_mode": "0", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T05:18:23.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T05:18:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T05:18:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T05:18:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_bits": "3", 2024-09-13T05:18:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T05:18:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_size": "10000", 2024-09-13T05:18:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T05:18:23.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_mid": "0.700000", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_trim_interval": "1", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T05:18:23.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_auth_pins": "false", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_frag": "false", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_scatterstat": "false", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_subtrees": "false", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_default_dir_hash": "2", 2024-09-13T05:18:23.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_defer_session_stale": "true", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_max_entries": "0", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_prefetch": "true", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_early_reply": "true", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_enable_op_tracker": "true", 2024-09-13T05:18:23.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_enforce_unique_name": "true", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T05:18:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_health_dummy": "false", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_join_fs": "", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_journal_format": "1", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_create_at": "0", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_export_at": "0", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_import_at": "0", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_journal_at": "0", 2024-09-13T05:18:23.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_link_at": "0", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_openc_at": "0", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_rename_at": "0", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_max_events": "-1", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_max_segments": "128", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_pause": "false", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_segment_size": "0", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T05:18:23.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_completed_requests": "100000", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_export_size": "20971520", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_file_recover": "32", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_purge_files": "64", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_purge_ops": "8192", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_metrics_update_interval": "2", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_min_caps_per_client": "100", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T05:18:23.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_numa_node": "-1", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_duration": "600", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_size": "20", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_log_threshold": "5", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_ping_grace": "15", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_ping_interval": "5", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_max_caps": "30000", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T05:18:23.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_replay_interval": "1.000000", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_root_ino_gid": "0", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_root_ino_uid": "0", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T05:18:23.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_shutdown_check": "0", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_skip_ino": "0", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_snap_min_uid": "0", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_snap_rstat": "false", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_symlink_recovery": "true", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T05:18:23.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_thrash_exports": "0", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_thrash_fragments": "0", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_tick_interval": "5.000000", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_valgrind_exit": "false", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_verify_backtrace": "1", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_verify_scatter": "false", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_wipe_sessions": "false", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool_debug": "false", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_page_set": "false", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_page_size": "65536", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_client_bytes": "134217728", 2024-09-13T05:18:23.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_client_messages": "512", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_disabled_modules": "", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mds_messages": "128", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mon_messages": "128", 2024-09-13T05:18:23.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_osd_messages": "8192", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_pool": "true", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_standby_modules": "true", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_stats_period": "5", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_stats_threshold": "5", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_tick_period": "2", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_allow_pool_delete": "true", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_allow_pool_size_one": "true", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T05:18:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_bytes": "104857600", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_directed_command_retry": "5", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T05:18:23.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_file_level": "debug", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_journald": "false", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T05:18:23.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_compact_on_start": "false", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_compact_on_trim": "true", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cpu_threads": "4", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data_avail_crit": "5", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data_avail_warn": "5", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_dump_json": "false", 2024-09-13T05:18:23.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_extra_checks": "false", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_require_quincy": "false", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_down_mkfs_grace": "120", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_election_default_strategy": "1", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_election_timeout": "5.000000", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T05:18:23.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_enable_op_tracker": "true", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_fake_pool_delete": "false", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_force_quorum_join": "false", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_detail_to_clog": "false", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_log_update_period": "5", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_max_detail": "50", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_to_clog": "true", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_host": "172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.79", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_host_override": "", 2024-09-13T05:18:23.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_initial_members": "", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_lease": "5.000000", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_log_full_interval": "50", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_log_max": "10000", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_log_max_summary": "50", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T05:18:23.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_log_epochs": "500", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_osd": "10000", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mds_skip_sanity": "false", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_memory_autotune": "true", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_memory_target": "2147483648", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_digest_period": "5", 2024-09-13T05:18:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_min_osdmap_epochs": "500", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_complaint_time": "30", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_duration": "600", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_size": "20", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_log_threshold": "5", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T05:18:23.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_cache_size": "500", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T05:18:23.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T05:18:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_report_timeout": "900", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T05:18:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_max_osds": "4", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_interval": "86400", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_max_keys": "100", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_timeout": "300", 2024-09-13T05:18:23.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_session_timeout": "300", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_smart_report_timeout": "5", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_pool_size": "4", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_debug": "false", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T05:18:23.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_tick_interval": "5", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T05:18:23.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_older_version": "true", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "monmap": "", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T05:18:23.779 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_tracing_enabled": "false", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_op_threads": "3", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_cm": "false", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_device_name": "", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T05:18:23.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_sl": "3", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_type": "ib", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_reap_threshold": "5", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_before_connect": "false", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_ipv4": "true", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_ipv6": "false", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_msgr1": "true", 2024-09-13T05:18:23.781 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_msgr2": "true", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_port_max": "7568", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_port_min": "6800", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_retry_count": "3", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_retry_delay": "5", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_client": "false", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_mds": "false", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_mgr": "false", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_mon": "false", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_osd": "false", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_client_mode": "crc secure", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T05:18:23.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_cluster_type": "", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_compress_secure": "false", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_crc_data": "true", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_crc_header": "true", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_bug": "true", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_old_message": "true", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T05:18:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T05:18:23.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T05:18:23.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T05:18:23.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T05:18:23.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T05:18:23.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T05:18:23.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T05:18:23.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T05:18:23.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_hugepages": "", 2024-09-13T05:18:23.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_lro": "true", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_pmd": "", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_port_id": "0", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dump_on_send": "false", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-13T05:18:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_delay_type": "", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_network_congestion": "0", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_socket_failures": "5000", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_max_accept_failures": "4", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_max_backoff": "15.000000", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_osd_compress_mode": "none", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T05:18:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_pq_min_cost": "65536", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_public_type": "", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_service_mode": "crc secure", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_nodelay": "true", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_type": "async+posix", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "no_config_file": "false", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_inflight_ops": "1024", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T05:18:23.787 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_timeout": "10.000000", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "objectstore_blackhole": "false", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "openssl_engine_opts": "", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_max_ops": "4", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_slop": "0.020000", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T05:18:23.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_asio_thread_count": "2", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_async_recovery_min_cost": "100", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backfill_scan_max": "512", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backfill_scan_min": "64", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backoff_on_peering": "false", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_beacon_report_interval": "300", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_duration": "30", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T05:18:23.789 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_blkin_trace_all": "false", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_update_on_start": "true", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_message_cap": "256", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_op_priority": "63", 2024-09-13T05:18:23.790 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_watch_timeout": "30", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_command_max_records": "256", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_compact_on_start": "false", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_update_on_start": "true", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T05:18:23.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_op_order": "false", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_shutdown": "false", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T05:18:23.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-13T05:18:23.793 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_default_notify_timeout": "30", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_enable_op_tracker": "true", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_info": "true", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_shutdown": "true", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T05:18:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_function_tracing": "false", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_grace": "20", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_interval": "6", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_stale": "600", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T05:18:23.795 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_journal_size": "5120", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_kill_backfill_at": "0", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_cache_size": "50", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_dedup": "true", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_message_max": "40", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T05:18:23.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_attr_name_len": "100", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_attr_size": "0", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_backfills": "1", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_markdown_count": "5", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_markdown_period": "600", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_object_name_len": "2048", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_object_size": "134217728", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T05:18:23.797 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pgls": "1024", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_push_cost": "8388608", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_push_objects": "10", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_scrubs": "3", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_write_size": "90", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T05:18:23.798 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_profile": "balanced", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T05:18:23.799 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_base": "805306368", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_target": "4294967296", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_target_autotune": "false", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_min_recovery_priority": "0", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_report_interval": "5", 2024-09-13T05:18:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_num_cache_shards": "32", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_numa_node": "-1", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objecter_finishers": "1", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objectstore": "bluestore", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objectstore_fuse": "false", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objectstore_tracing": "false", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_duration": "600", 2024-09-13T05:18:23.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_size": "20", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_log_threshold": "5", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_shards": "0", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T05:18:23.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_thread_timeout": "15", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_open_classes_on_start": "true", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_os_flags": "0", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_peering_op_priority": "255", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_delete_priority": "5", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T05:18:23.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_crimson": "false", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T05:18:23.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flags": "0", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_min_size": "0", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_size": "2", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_type": "replicated", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T05:18:23.805 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_cost": "20971520", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_active": "0", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_op_priority": "3", 2024-09-13T05:18:23.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_priority": "5", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_repair_during_recovery": "false", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T05:18:23.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_cost": "52428800", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_end_hour": "0", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T05:18:23.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_priority": "5", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_reservation_timeout": "5000", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_skip_data_digest": "false", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_smart_report_timeout": "5", 2024-09-13T05:18:23.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_priority": "5", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_target_transaction_size": "30", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T05:18:23.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tracing": "false", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_use_stale_snap": "false", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_kill_at": "0", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_max_join_drift": "10", 2024-09-13T05:18:23.811 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_min": "500", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_min_wait": "0.050000", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_service_trim_max": "500", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_service_trim_min": "250", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_stash_full_interval": "25", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_trim_max": "500", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_trim_min": "250", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "perf": "true", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-13T05:18:23.812 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "-", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrv": "", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "public_bind_addr": "-", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "public_network": "", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "public_network_interface": "", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "qat_compressor_enabled": "false", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "rados_mon_op_timeout": "0", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "rados_osd_op_timeout": "0", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "rados_tracing": "false", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_atime_update_interval": "60", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T05:18:23.813 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache": "true", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_policy": "writearound", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_size": "33554432", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T05:18:23.814 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_compression_hint": "none", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_clone_format": "auto", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_data_pool": "", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_features": "61", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_format": "2", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_map_options": "", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_order": "22", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_pool": "rbd", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T05:18:23.815 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_stripe_count": "0", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_io_scheduler": "simple", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T05:18:23.816 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_order": "24", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_pool": "", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_splay_width": "4", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T05:18:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T05:18:23.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T05:18:23.819 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_op_threads": "1", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T05:18:23.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_plugins": "", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T05:18:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T05:18:23.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_tracing": "false", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_validate_names": "true", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_validate_pool": "true", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "restapi_base_url": "", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "restapi_log_level": "", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_admin_entry": "admin", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_backend_store": "rados", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_barbican_url": "", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_beast_enable_async": "true", 2024-09-13T05:18:23.823 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cache_enabled": "true", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cache_lru_size": "10000", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_config_store": "rados", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_content_length_compat": "false", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T05:18:23.824 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cross_domain_policy": "", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T05:18:23.825 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T05:18:23.826 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T05:18:23.827 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T05:18:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_window": "30", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T05:18:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T05:18:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dns_name": "", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dns_s3website_name": "", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_ops_log": "false", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_static_website": "false", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_usage_log": "false", 2024-09-13T05:18:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_expose_bucket": "false", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_extended_http_attrs": "", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_filter": "none", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_objs": "32", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T05:18:23.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_init_timeout": "300", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_password": "", 2024-09-13T05:18:23.833 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_project": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_token": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_user": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_api_version": "2", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T05:18:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_url": "", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_objs": "32", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_worker": "3", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T05:18:23.835 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_uri": "ldaps://", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_http_headers": "", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_attr_size": "0", 2024-09-13T05:18:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_listing_results": "1000", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_notify_retries": "10", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T05:18:23.837 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T05:18:23.838 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_num_control_oids": "8", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_numa_node": "-1", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T05:18:23.839 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_op_tracing": "false", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_opa_token": "", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_opa_url": "", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_rados": "false", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T05:18:23.840 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_print_continue": "true", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rados_tracing": "false", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_realm": "", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_realm_id": "", 2024-09-13T05:18:23.841 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_region": "", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_request_uri": "", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_resolve_cname": "false", 2024-09-13T05:18:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_run_sync_thread": "true", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_script_uri": "", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_service_provider_name": "", 2024-09-13T05:18:23.843 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_client_id": "", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_client_secret": "", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_entry": "sts", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_key": "sts", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_auth_url": "", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_custom_header": "", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_need_stats": "true", 2024-09-13T05:18:23.844 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_tenant_name": "", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_url": "", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_lease_period": "120", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T05:18:23.845 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_thread_pool_size": "512", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_comment": "", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_createby": "", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_encoding": "", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_flag": "false", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_origin": "", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_tracker": "", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T05:18:23.846 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_max_shards": "32", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_use_opa_authz": "false", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_unique_email": "true", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_verify_ssl": "true", 2024-09-13T05:18:23.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zone": "", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zone_id": "", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zonegroup": "", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zonegroup_id": "", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_block_size": "4096", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T05:18:23.848 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T05:18:23.849 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_partition_filters": "false", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_perf": "false", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "run_dir": "/var/run/ceph", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_block_create": "true", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_device_size": "53687091200", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T05:18:23.850 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_main_device_type": "SSD", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_segment_size": "67108864", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "set_keepcaps": "false", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "setgroup": "", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "setuser": "", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "setuser_match_path": "", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T05:18:23.851 INFO:teuthology.orchestra.run.smithi016.stdout: "thp": "false", 2024-09-13T05:18:23.852 INFO:teuthology.orchestra.run.smithi016.stdout: "threadpool_default_timeout": "60", 2024-09-13T05:18:23.852 INFO:teuthology.orchestra.run.smithi016.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T05:18:23.852 INFO:teuthology.orchestra.run.smithi016.stdout: "throttler_perf_counter": "true" 2024-09-13T05:18:23.852 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:23.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T05:18:23.999 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a: {} 2024-09-13T05:18:24.000 INFO:teuthology.orchestra.run.smithi016.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T05:18:24.013 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b: {} 2024-09-13T05:18:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T05:18:24.023 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c: {} 2024-09-13T05:18:24.023 INFO:teuthology.orchestra.run.smithi016.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T05:18:24.040 DEBUG:teuthology.orchestra.run.smithi079:> type ceph-objectstore-tool 2024-09-13T05:18:24.045 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T05:18:24.046 DEBUG:teuthology.orchestra.run.smithi016:> type ceph-objectstore-tool 2024-09-13T05:18:24.050 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T05:18:24.050 INFO:teuthology.run_tasks:Running task workunit... 2024-09-13T05:18:24.064 INFO:tasks.workunit:Pulling workunits from ref 7a853b1706d076d58e0da7a5d3b9b9eb36276880 2024-09-13T05:18:24.064 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T05:18:24.065 DEBUG:teuthology.orchestra.run.smithi122:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T05:18:24.065 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T05:18:24.065 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-13T05:18:24.065 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:18:24.066 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5] 2024-09-13T05:18:24.066 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T05:18:24.066 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:18:24.066 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T05:18:24.066 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T05:18:24.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:18:24.067 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T05:18:24.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:24.067 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T05:18:24.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:18:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:18:24.073 INFO:teuthology.orchestra.run.smithi122.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T05:18:24.074 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T05:18:24.094 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:24.091+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:24.119 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T05:18:24.119 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T05:18:24.161 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T05:18:24.162 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-09-13T05:18:24.162 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:18:24.162 DEBUG:teuthology.orchestra.run.smithi079:> type ceph-objectstore-tool 2024-09-13T05:18:24.167 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T05:18:24.167 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-09-13T05:18:24.168 INFO:tasks.workunit:timeout=3h 2024-09-13T05:18:24.168 INFO:tasks.workunit:cleanup=True 2024-09-13T05:18:24.169 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7a853b1706d076d58e0da7a5d3b9b9eb36276880 2024-09-13T05:18:24.169 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:24.163+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:24.214 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:24.214 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:24.217 INFO:tasks.workunit.client.0.smithi122.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T05:18:24.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:24.234 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:24.252 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:24.252 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:24.268 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:24.268 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:24.268 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-13T05:18:24.269 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:24.267+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:24.286 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:24.286 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:24.302 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:24.302 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:18:24.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:18:24.343 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:24.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:24.406 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:24.403+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:24.471 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:24.467+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:24.516 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:24.516 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:24.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:24.572 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:24.572+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:24.674 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:24.676+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:24.699 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:24.700 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:24.714 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:24.774 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:24.776+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:24.834 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:24.834 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:24.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:24.897 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:24.895+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:24.963 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:24.963 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:24.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:24.976 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:24.971+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:25.077 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:25.076+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:25.127 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:25.127 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:25.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:25.159 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:18:25.177 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:25.175+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:25.261 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:25.261 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:25.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:25.317 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:25.315+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:25.379 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:25.375+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:25.388 INFO:teuthology.orchestra.run.smithi079.stdout:2.4 2024-09-13T05:18:25.388 INFO:teuthology.orchestra.run.smithi079.stdout:2.1 2024-09-13T05:18:25.398 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:25.398 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:25.412 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:25.480 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:25.475+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:25.533 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:25.533 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:25.547 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:25.594 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:25.596+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:25.662 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:25.662 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:25.675 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:25.681 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:25.679+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:25.782 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:25.779+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:25.841 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:25.841 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:25.855 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:25.902 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:25.904+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:25.959 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 2.4 --file /var/log/ceph/exp.2.4.3 2024-09-13T05:18:25.972 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:25.972 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:25.984 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:25.979+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:25.986 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:26.086 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:26.084+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:26.146 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:26.146 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:26.160 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:26.202 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:26.204+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:26.287 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:26.283+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:26.306 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:18:26.327 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:26.328 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:26.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:26.388 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:26.388+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:26.462 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:26.478 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:26.479 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:26.480 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:26.480 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:26.480 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:26.480 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:26.489 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:26.487+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:26.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:26.495 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-13T05:18:26.516 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:26.535 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:18:26.540 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:18:26.590 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:26.592+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:26.610 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:26.610 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:26.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:26.691 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:26.687+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:26.748 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.611089145999999, 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:26.749 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.610449177, 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:26.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:26.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:26.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:26.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:18:26.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.754 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:26.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:26.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.6155761159999997, 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:18:26.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:26.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:18:26.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.6094643239999993, 2024-09-13T05:18:26.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:26.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.605794586, 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:26.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:18:26.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:26.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.5996568670000002, 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:26.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:18:26.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:18:26.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:26.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:26.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:26.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:18:26.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:18:26.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.779 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:26.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:26.779 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:26.779 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:26.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:26.821 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:26.819+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:26.893 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:26.892+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:26.950 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:26.951 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:26.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:27.013 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:27.011+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:27.086 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:27.086 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:27.094 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:27.096+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:27.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:27.124 INFO:teuthology.orchestra.run.smithi079.stderr:Exporting 2.4 info 2.4( empty local-lis/les=16/17 n=0 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) 2024-09-13T05:18:27.126 INFO:teuthology.orchestra.run.smithi079.stderr:Export successful 2024-09-13T05:18:27.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:18:27.233 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:27.231+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:27.269 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.409826034999998, 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010082459, 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858216+0000", 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858229+0000", 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.305e-05 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858266+0000", 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7523e-05 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858283+0000", 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6597000000000001e-05 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865356+0000", 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070732859999999998 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865490+0000", 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133442 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865698+0000", 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208202 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867616+0000", 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019181409999999999 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868169+0000", 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055310299999999999 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868246+0000", 2024-09-13T05:18:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6866000000000006e-05 2024-09-13T05:18:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868300+0000", 2024-09-13T05:18:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3912999999999997e-05 2024-09-13T05:18:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.277 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.277 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:27.277 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.409737829000001, 2024-09-13T05:18:27.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010336577, 2024-09-13T05:18:27.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858304+0000", 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858318+0000", 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3963e-05 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858337+0000", 2024-09-13T05:18:27.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8482999999999999e-05 2024-09-13T05:18:27.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858351+0000", 2024-09-13T05:18:27.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4195999999999999e-05 2024-09-13T05:18:27.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866611+0000", 2024-09-13T05:18:27.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082601199999999993 2024-09-13T05:18:27.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866763+0000", 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015145600000000001 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866987+0000", 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224441 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867784+0000", 2024-09-13T05:18:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079658099999999998 2024-09-13T05:18:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868531+0000", 2024-09-13T05:18:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074704700000000003 2024-09-13T05:18:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868596+0000", 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4966000000000001e-05 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868642+0000", 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6607000000000003e-05 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.315839267000001, 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.285 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:27.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:27.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:18:27.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:18:27.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:18:27.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:18:27.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:18:27.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:18:27.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:18:27.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:18:27.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.292 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.315632108999999, 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:18:27.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:18:27.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:18:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.13839338, 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010071422, 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.299 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 107 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129649+0000", 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129656+0000", 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129668+0000", 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2463000000000001e-05 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129675+0000", 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1999999999999997e-06 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131247+0000", 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015719359999999999 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131321+0000", 2024-09-13T05:18:27.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4232000000000003e-05 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131413+0000", 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1525000000000002e-05 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.133468+0000", 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002055177 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139645+0000", 2024-09-13T05:18:27.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061772490000000001 2024-09-13T05:18:27.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139691+0000", 2024-09-13T05:18:27.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5967000000000002e-05 2024-09-13T05:18:27.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139722+0000", 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0273e-05 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.137649154999998, 2024-09-13T05:18:27.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:18:27.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:18:27.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:18:27.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:18:27.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:18:27.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:18:27.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:18:27.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.312 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.137520283000001, 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.313 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:18:27.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:18:27.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.319 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.137234991, 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010554714999999999, 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 115 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130808+0000", 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130882+0000", 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4449999999999994e-05 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130891+0000", 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7190000000000007e-06 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130897+0000", 2024-09-13T05:18:27.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:18:27.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135420+0000", 2024-09-13T05:18:27.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045232060000000001 2024-09-13T05:18:27.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135472+0000", 2024-09-13T05:18:27.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1530999999999998e-05 2024-09-13T05:18:27.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135557+0000", 2024-09-13T05:18:27.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5192999999999996e-05 2024-09-13T05:18:27.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136761+0000", 2024-09-13T05:18:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001204072 2024-09-13T05:18:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141283+0000", 2024-09-13T05:18:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045216099999999997 2024-09-13T05:18:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141331+0000", 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8094999999999999e-05 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141363+0000", 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2319e-05 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.137140723999998, 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010559903000000001, 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 116 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.327 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130902+0000", 2024-09-13T05:18:27.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130905+0000", 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130909+0000", 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7239999999999998e-06 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130914+0000", 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1800000000000004e-06 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135956+0000", 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050418850000000003 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136008+0000", 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1694999999999999e-05 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136090+0000", 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2527000000000005e-05 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137204+0000", 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011136640000000001 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141406+0000", 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042020039999999996 2024-09-13T05:18:27.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141438+0000", 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2277999999999998e-05 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141463+0000", 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4396000000000002e-05 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.332 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.333 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.333 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:27.333 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.137122868999999, 2024-09-13T05:18:27.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:18:27.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.334 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.334 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.334 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:18:27.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:18:27.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:18:27.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.339 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.752742546, 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:18:27.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.752705410000001, 2024-09-13T05:18:27.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:18:27.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:18:27.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:18:27.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:18:27.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:18:27.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:18:27.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:18:27.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:18:27.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:18:27.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:18:27.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:18:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:18:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:27.355 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.752599491000002, 2024-09-13T05:18:27.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:18:27.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:27.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:18:27.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:27.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:18:27.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:18:27.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:18:27.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:18:27.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:18:27.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:18:27.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:18:27.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:18:27.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:18:27.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:18:27.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:18:27.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:18:27.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:18:27.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.752327466000001, 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:18:27.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:18:27.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:18:27.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.75228388, 2024-09-13T05:18:27.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:18:27.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:18:27.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:18:27.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:18:27.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:18:27.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.752251652999998, 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:18:27.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:27.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:18:27.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:18:27.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:18:27.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.752040505, 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:18:27.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:27.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:18:27.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:18:27.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:18:27.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:18:27.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:18:27.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:18:27.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:18:27.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:18:27.389 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.389 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.390 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.390 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:27.390 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.752004211999999, 2024-09-13T05:18:27.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:27.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:27.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:18:27.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:18:27.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:18:27.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:18:27.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:18:27.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.397 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.751965762000001, 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.398 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:18:27.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:18:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:18:27.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:18:27.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:18:27.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:27.404 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.75179086, 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:18:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:18:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:27.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:18:27.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.410 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:27.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:27.413 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:27.412+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:27.513 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:27.511+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:27.591 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:27.591 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:27.592 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:27.592 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:27.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:27.614 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:27.616+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:27.662 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 2.4 2024-09-13T05:18:27.750 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:27.752+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:27.775 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:27.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:27.776 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:27.776 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:27.776 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:27.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:27.837 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:27.835+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:27.909 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:27.909 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:27.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:27.909 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:27.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.777547019, 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:18:27.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:27.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:27.914 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:27.916 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:27.911+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:27.924 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:28.017 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:28.016+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:28.055 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:28.056 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:28.071 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:28.158 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:28.160+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:18:28.219 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:28.220+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:28.227 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:28.228 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:28.228 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:28.228 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:28.228 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:28.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:28.320 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:28.315+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 11.270898938, 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001669217, 2024-09-13T05:18:28.408 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:28.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139957+0000", 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139968+0000", 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.007e-05 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139993+0000", 2024-09-13T05:18:28.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5945999999999999e-05 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140008+0000", 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4926000000000001e-05 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140092+0000", 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3282999999999995e-05 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140137+0000", 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5195999999999999e-05 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141553+0000", 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014160659999999999 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141627+0000", 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4330000000000002e-05 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 10.261194743000001, 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001729281, 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:28.413 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:28.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149679+0000", 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7090000000000001e-05 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149706+0000", 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7192999999999999e-05 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149721+0000", 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.45e-05 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149824+0000", 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000102959 2024-09-13T05:18:28.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149862+0000", 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8667000000000001e-05 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151293+0000", 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001431332 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:28.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151392+0000", 2024-09-13T05:18:28.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8023000000000007e-05 2024-09-13T05:18:28.418 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:28.418 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:28.418 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:28.418 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:28.418 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:28.418 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:28.420 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:28.420+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:28.423 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:28.521 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:28.524+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:28.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:18:28.597 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:28.597 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:28.597 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:28.597 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:28.611 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:28.654 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:28.656+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:28.699 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:28.716 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:28.716 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:28.723 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:28.724+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:28.732 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:28.732 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:28.733 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-13T05:18:28.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:28.773 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:28.773 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:28.774 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:28.774 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:28.777 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:18:28.778 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:18:28.787 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:28.830 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:28.832+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:28.893 INFO:teuthology.orchestra.run.smithi079.stdout: marking collection for removal 2024-09-13T05:18:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:setting '_remove' omap key 2024-09-13T05:18:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:finish_remove_pgs 2.4_head removing 2.4 2024-09-13T05:18:28.896 INFO:teuthology.orchestra.run.smithi079.stdout:Remove successful 2024-09-13T05:18:28.925 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:28.924+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:28.952 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:28.952 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:28.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:29.026 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:29.023+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:29.101 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:29.102 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:29.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:29.165 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:29.163+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:29.228 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:29.228+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:29.295 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:29.296 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:29.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:29.357 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:29.355+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:29.428 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.4.3 2024-09-13T05:18:29.429 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:29.429 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:29.430 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:29.427+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:29.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:29.531 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:29.527+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:29.603 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:29.604 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:29.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:29.632 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:29.632+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:29.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:29.731+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:29.788 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:29.788 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:29.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:29.833 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:29.836+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:29.935 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:29.931+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:29.964 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:29.964 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:29.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:30.036 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:30.036+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:30.137 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:30.135+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:30.147 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:30.147 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:30.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:30.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:18:30.238 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:30.240+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:30.331 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:30.332 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:30.338 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:30.335+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:30.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:30.439 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:30.440+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:30.504 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:30.505 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:30.519 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:30.566 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:30.568+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:30.641 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:30.639+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:30.643 INFO:teuthology.orchestra.run.smithi079.stdout:get_pg_num_history pg_num_history pg_num_history(e20 pg_nums {1={13=1},2={16=8}} deleted_pools ) 2024-09-13T05:18:30.644 INFO:teuthology.orchestra.run.smithi079.stdout:Importing pgid 2.4 2024-09-13T05:18:30.644 INFO:teuthology.orchestra.run.smithi079.stdout:write_pg epoch 20 info 2.4( empty local-lis/les=16/17 n=0 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) 2024-09-13T05:18:30.645 INFO:teuthology.orchestra.run.smithi079.stdout:Import successful 2024-09-13T05:18:30.653 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:30.654 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:30.667 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:30.758 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:30.760+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:30.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:18:30.831 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:30.831 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:30.843 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:30.844+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:30.845 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:30.932 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:30.943 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:30.939+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:30.948 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:30.964 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-13T05:18:30.983 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:31.005 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:18:31.008 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:18:31.016 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:31.016 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:31.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:31.016 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:31.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 13.879519413000001, 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001669217, 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139957+0000", 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139968+0000", 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.007e-05 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139993+0000", 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5945999999999999e-05 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140008+0000", 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4926000000000001e-05 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140092+0000", 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3282999999999995e-05 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140137+0000", 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5195999999999999e-05 2024-09-13T05:18:31.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141553+0000", 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014160659999999999 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141627+0000", 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4330000000000002e-05 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 12.869815217999999, 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001729281, 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149679+0000", 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7090000000000001e-05 2024-09-13T05:18:31.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149706+0000", 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7192999999999999e-05 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149721+0000", 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.45e-05 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149824+0000", 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000102959 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149862+0000", 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8667000000000001e-05 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151293+0000", 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001431332 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151392+0000", 2024-09-13T05:18:31.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8023000000000007e-05 2024-09-13T05:18:31.026 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.026 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.026 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.026 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.026 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.026 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:31.032 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:31.049 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:31.047+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:31.145 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:31.143+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:31.151 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:31.151 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:31.152 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:31.152 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:31.165 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:31.184 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /var/log/ceph/exp.2.4.3 2024-09-13T05:18:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:18:31.253 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:31.251+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:31.288 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:31.289 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:31.303 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:31.346 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:31.343+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:31.426 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:31.426 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:31.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:31.426 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:31.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'53)", 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.464547064000001, 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050448339999999998, 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961433+0000", 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961492+0000", 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9488999999999999e-05 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961517+0000", 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5296999999999999e-05 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961529+0000", 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1763000000000001e-05 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961621+0000", 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1636000000000002e-05 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961656+0000", 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5363e-05 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966359+0000", 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047030839999999997 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966479+0000", 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119839 2024-09-13T05:18:31.430 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.293455307999999, 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:18:31.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.435 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.292824049, 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:18:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.439 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.292178927999998, 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:18:31.440 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:18:31.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:18:31.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.291671565000001, 2024-09-13T05:18:31.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:31.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.447 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:18:31.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.291146129000001, 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:18:31.450 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:18:31.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.290508617, 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:18:31.455 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.456 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.456 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:31.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:18:31.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.458 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.289931909, 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.460 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.462 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:18:31.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.464 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.289380435999998, 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:18:31.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:18:31.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.288919852999999, 2024-09-13T05:18:31.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.471 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:18:31.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.288409967, 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:18:31.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:18:31.478 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.287982904, 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:31.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.480 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:18:31.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:18:31.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:18:31.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:18:31.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:116 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'78, mlcod=14'65)", 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.287522216999999, 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050255040000000001, 2024-09-13T05:18:31.483 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138458+0000", 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138469+0000", 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1596e-05 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138490+0000", 2024-09-13T05:18:31.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0446e-05 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138521+0000", 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0843000000000003e-05 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142091+0000", 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035703929999999998 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142146+0000", 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4873000000000001e-05 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143395+0000", 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012491169999999999 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143485+0000", 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9419000000000006e-05 2024-09-13T05:18:31.487 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:117 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'79, mlcod=14'65)", 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 21.287030508000001, 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050956009999999999, 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138949+0000", 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138962+0000", 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2876e-05 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138987+0000", 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4652999999999999e-05 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139021+0000", 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4122999999999998e-05 2024-09-13T05:18:31.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142737+0000", 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003715472 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142773+0000", 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6086e-05 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144004+0000", 2024-09-13T05:18:31.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001231631 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144046+0000", 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1996000000000002e-05 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:132 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'90, mlcod=14'86)", 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:31.492 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 20.903843242000001, 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005226835, 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522137+0000", 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522142+0000", 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7040000000000002e-06 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522153+0000", 2024-09-13T05:18:31.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0573e-05 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522158+0000", 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9389999999999996e-06 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522191+0000", 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3571000000000001e-05 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522204+0000", 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3169999999999999e-05 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527326+0000", 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00512153 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527365+0000", 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8971999999999999e-05 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.496 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 20.901223185999999, 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:31.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:153 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'107, mlcod=14'105)", 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 20.284896485000001, 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046950569999999999, 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.501 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141083+0000", 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141104+0000", 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0945999999999999e-05 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141127+0000", 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3054e-05 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141149+0000", 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1690000000000001e-05 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141266+0000", 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000117359 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141303+0000", 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7013e-05 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145722+0000", 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044190260000000004 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145780+0000", 2024-09-13T05:18:31.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7751999999999999e-05 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 20.283688374, 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.510 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:156 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'110, mlcod=14'105)", 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 20.283109501999999, 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047886200000000004, 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142871+0000", 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142890+0000", 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9882999999999999e-05 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142944+0000", 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3465999999999999e-05 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142959+0000", 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4644000000000001e-05 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144037+0000", 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001077988 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144094+0000", 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7079e-05 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147601+0000", 2024-09-13T05:18:31.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035074400000000001 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147660+0000", 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9243000000000002e-05 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:159 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'113, mlcod=14'105)", 2024-09-13T05:18:31.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 20.281424401999999, 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044745929999999998, 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:31.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144556+0000", 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144583+0000", 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7416e-05 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:31.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144632+0000", 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9453999999999998e-05 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144651+0000", 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8635999999999999e-05 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.146970+0000", 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023192159999999998 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147003+0000", 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2595999999999998e-05 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.148955+0000", 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001951595 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.149031+0000", 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6922999999999995e-05 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.519 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.520 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.520 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:31.520 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:31.520 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:31.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:31.522 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:31.519+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:31.622 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:31.624+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:31.657 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:31.658 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:31.658 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:31.658 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:31.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:31.722 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:31.719+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:31.797 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:31.797 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:31.797 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:31.797 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:31.797 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:31.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:31.853 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:31.851+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:31.924 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:31.919+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:31.981 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:31.981 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 15.843818927999999, 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:31.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:18:31.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:31.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 15.843178958999999, 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:31.986 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:31.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:18:31.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:31.990 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.848305898, 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:18:31.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:31.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.997 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.842194106000001, 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:31.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:18:31.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:18:32.001 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.838524368, 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:18:32.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:18:32.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:18:32.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:32.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.008 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.832386649, 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:18:32.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.012 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:32.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:32.057 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:32.055+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:32.126 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:32.123+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:32.178 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:32.178 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:32.178 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:32.178 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:32.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:32.227 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:32.228+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:32.328 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:32.323+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:32.364 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:32.364 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:32.365 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:32.365 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:32.365 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:32.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:32.465 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:32.463+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:32.530 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:32.527+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:32.549 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:32.549 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:32.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.689878296, 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010082459, 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858216+0000", 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858229+0000", 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.305e-05 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858266+0000", 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7523e-05 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858283+0000", 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6597000000000001e-05 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865356+0000", 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070732859999999998 2024-09-13T05:18:32.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865490+0000", 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133442 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865698+0000", 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208202 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867616+0000", 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019181409999999999 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868169+0000", 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055310299999999999 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868246+0000", 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6866000000000006e-05 2024-09-13T05:18:32.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868300+0000", 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3912999999999997e-05 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.689790089999999, 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010336577, 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858304+0000", 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858318+0000", 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3963e-05 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858337+0000", 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8482999999999999e-05 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858351+0000", 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4195999999999999e-05 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866611+0000", 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082601199999999993 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866763+0000", 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015145600000000001 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866987+0000", 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224441 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867784+0000", 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079658099999999998 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868531+0000", 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074704700000000003 2024-09-13T05:18:32.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868596+0000", 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4966000000000001e-05 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868642+0000", 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6607000000000003e-05 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.595891527999999, 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.563 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:32.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:18:32.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.595684370000001, 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.570 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:32.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:18:32.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.418445641000002, 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010071422, 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.576 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 107 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129649+0000", 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129656+0000", 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2024-09-13T05:18:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129668+0000", 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2463000000000001e-05 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129675+0000", 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1999999999999997e-06 2024-09-13T05:18:32.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131247+0000", 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015719359999999999 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131321+0000", 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4232000000000003e-05 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131413+0000", 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1525000000000002e-05 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.133468+0000", 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002055177 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139645+0000", 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061772490000000001 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139691+0000", 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5967000000000002e-05 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139722+0000", 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0273e-05 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.582 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.417701416, 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:18:32.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.417572543999999, 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:18:32.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.417287252000001, 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010554714999999999, 2024-09-13T05:18:32.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 115 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130808+0000", 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130882+0000", 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4449999999999994e-05 2024-09-13T05:18:32.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130891+0000", 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7190000000000007e-06 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130897+0000", 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135420+0000", 2024-09-13T05:18:32.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045232060000000001 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135472+0000", 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1530999999999998e-05 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135557+0000", 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5192999999999996e-05 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136761+0000", 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001204072 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141283+0000", 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045216099999999997 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141331+0000", 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8094999999999999e-05 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141363+0000", 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2319e-05 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.417192985, 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010559903000000001, 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 116 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130902+0000", 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130905+0000", 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130909+0000", 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7239999999999998e-06 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130914+0000", 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1800000000000004e-06 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135956+0000", 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050418850000000003 2024-09-13T05:18:32.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136008+0000", 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1694999999999999e-05 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136090+0000", 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2527000000000005e-05 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137204+0000", 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011136640000000001 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141406+0000", 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042020039999999996 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141438+0000", 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2277999999999998e-05 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141463+0000", 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4396000000000002e-05 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.41717513, 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:18:32.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:18:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.032794806999998, 2024-09-13T05:18:32.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:18:32.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.032757670999999, 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.625 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.032651752, 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:18:32.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:18:32.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:18:32.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.032379726999999, 2024-09-13T05:18:32.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:18:32.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:32.638 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.032336140999998, 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:32.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:18:32.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.032303914, 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:18:32.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:18:32.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.650 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.032092766000002, 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:18:32.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.656 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.032056473000001, 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:18:32.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.662 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.032018022999999, 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:18:32.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:18:32.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.668 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.031843121000001, 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:32.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:32.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:32.674 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:32.675 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:32.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:32.677 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:32.675+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:32.778 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:32.780+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:32.846 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:32.846 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:32.847 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:32.847 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:32.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:32.879 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:32.880+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:32.980 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:32.980+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:32.988 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:32.988 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:33.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:33.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:18:33.081 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:33.080+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:33.130 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:33.130 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:33.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.998775539, 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:33.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:33.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:18:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:33.135 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:33.146 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:33.194 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:33.194 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:33.195 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:33.196+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:33.211 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:33.211 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:33.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:18:33.227 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:33.228 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:33.228 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-13T05:18:33.249 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:33.270 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:33.273 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:18:33.274 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:18:33.280 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:33.280 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:33.283 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:33.284+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:33.293 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:33.384 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:33.379+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:33.452 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:33.452 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:33.466 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:33.510 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:33.512+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:33.581 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:33.581 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:33.586 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:33.588+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:33.595 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:33.687 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:33.688+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:33.764 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:33.764 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:33.778 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:33.788 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:33.783+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:33.889 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:33.883+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:33.944 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:33.944 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:33.959 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:34.006 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:34.008+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:34.083 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:34.083 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:34.090 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:34.087+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:34.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:34.191 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:34.192+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:34.216 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:18:34.233 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:34.234 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:34.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:34.291 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:34.287+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:34.392 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:34.387+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:34.413 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:34.413 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:34.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:34.493 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:34.496+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:34.594 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:34.594 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:34.594 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:34.596+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:34.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:34.695 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:34.691+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:34.733 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:34.734 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:34.748 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:34.796 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:34.796+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:34.897 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:34.896+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:34.916 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:34.916 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:34.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:34.998 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:35.000+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:35.050 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:35.050 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:35.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:35.099 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:35.100+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:35.185 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:35.185 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:35.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:35.200 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:35.195+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:35.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:18:35.302 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:35.304+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:35.318 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:35.319 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:35.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:35.402 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:35.404+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:35.424 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:35.424 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:35.442 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:35.442 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:35.458 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-13T05:18:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:35.479 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:35.500 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:35.500 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:35.503 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:35.504+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:18:35.504 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.372822620000001, 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:18:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:35.510 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:35.518 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:35.604 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:35.599+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:35.650 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:35.650 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:35.651 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:35.651 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:35.664 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:35.705 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:35.703+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:35.780 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:35.780 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:35.781 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:35.781 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:35.781 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:35.795 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:35.842 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:35.844+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:35.907 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:35.903+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:35.917 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:35.917 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:35.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:35.917 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:35.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.917 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-13T05:18:35.917 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:35.917 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.780366708999999, 2024-09-13T05:18:35.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001669217, 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:35.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139957+0000", 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139968+0000", 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.007e-05 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:35.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139993+0000", 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5945999999999999e-05 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140008+0000", 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4926000000000001e-05 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140092+0000", 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3282999999999995e-05 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140137+0000", 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5195999999999999e-05 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141553+0000", 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014160659999999999 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:35.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141627+0000", 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4330000000000002e-05 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 17.770662514000001, 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001729281, 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149679+0000", 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7090000000000001e-05 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149706+0000", 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7192999999999999e-05 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149721+0000", 2024-09-13T05:18:35.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.45e-05 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149824+0000", 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000102959 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149862+0000", 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8667000000000001e-05 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:35.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151293+0000", 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001431332 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151392+0000", 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8023000000000007e-05 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:35.926 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:35.931 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:36.022 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:36.024+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:36.098 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:36.098 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:36.098 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:36.098 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:36.109 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:36.108+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:36.112 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:36.210 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:36.207+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:36.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:18:36.240 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2024-09-13T05:18:36.240 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:18:36.240 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-13T05:18:36.240 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T05:18:36.240 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T05:18:36.242 INFO:tasks.ceph.osd.3:Started 2024-09-13T05:18:36.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:18:36.283 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:36.283 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:36.283 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:36.284 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:36.284 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:36.300 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:36.342 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:36.344+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:36.359 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-09-13T05:18:36.359 INFO:teuthology.orchestra.run.smithi079.stderr:0 2024-09-13T05:18:36.359 INFO:teuthology.orchestra.run.smithi079.stderr:1 2024-09-13T05:18:36.359 INFO:teuthology.orchestra.run.smithi079.stderr:2 2024-09-13T05:18:36.359 INFO:teuthology.orchestra.run.smithi079.stderr:abort 2024-09-13T05:18:36.360 INFO:teuthology.orchestra.run.smithi079.stderr:assert 2024-09-13T05:18:36.360 INFO:teuthology.orchestra.run.smithi079.stderr:config diff 2024-09-13T05:18:36.360 INFO:teuthology.orchestra.run.smithi079.stderr:config diff get 2024-09-13T05:18:36.360 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-09-13T05:18:36.360 INFO:teuthology.orchestra.run.smithi079.stderr:config help [] 2024-09-13T05:18:36.360 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-09-13T05:18:36.360 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: invalid command 2024-09-13T05:18:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T05:18:36.362 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-13T05:18:36.411 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:36.407+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:36.461 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:36.461 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'53)", 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.499953674, 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050448339999999998, 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.462 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961433+0000", 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961492+0000", 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9488999999999999e-05 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961517+0000", 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5296999999999999e-05 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961529+0000", 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1763000000000001e-05 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961621+0000", 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1636000000000002e-05 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961656+0000", 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5363e-05 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966359+0000", 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047030839999999997 2024-09-13T05:18:36.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966479+0000", 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119839 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.328861918000001, 2024-09-13T05:18:36.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:18:36.467 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.467 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.467 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.467 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.467 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:36.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:18:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:18:36.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:18:36.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:18:36.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:18:36.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.328230658999999, 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:18:36.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.327585538000001, 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.476 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:18:36.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.480 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.327078175, 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:18:36.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.326552739, 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:18:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:36.489 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.325915226999999, 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.493 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.325338518999999, 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:36.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:18:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.324787046000001, 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.498 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:18:36.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:18:36.502 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.324326462999998, 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:36.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.506 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.323816576999999, 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:36.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.323389513999999, 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.511 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:18:36.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:116 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'78, mlcod=14'65)", 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.322928826999998, 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050255040000000001, 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138458+0000", 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138469+0000", 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1596e-05 2024-09-13T05:18:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138490+0000", 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0446e-05 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138521+0000", 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0843000000000003e-05 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142091+0000", 2024-09-13T05:18:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035703929999999998 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142146+0000", 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4873000000000001e-05 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143395+0000", 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012491169999999999 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143485+0000", 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9419000000000006e-05 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:117 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'79, mlcod=14'65)", 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.322437118, 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050956009999999999, 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.520 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138949+0000", 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138962+0000", 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2876e-05 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138987+0000", 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4652999999999999e-05 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139021+0000", 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4122999999999998e-05 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142737+0000", 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003715472 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142773+0000", 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6086e-05 2024-09-13T05:18:36.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144004+0000", 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001231631 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144046+0000", 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1996000000000002e-05 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:132 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'90, mlcod=14'86)", 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 25.939249852, 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005226835, 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522137+0000", 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522142+0000", 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7040000000000002e-06 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522153+0000", 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0573e-05 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522158+0000", 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9389999999999996e-06 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522191+0000", 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3571000000000001e-05 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522204+0000", 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3169999999999999e-05 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527326+0000", 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00512153 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527365+0000", 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8971999999999999e-05 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.528 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 25.936629795999998, 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:36.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:18:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:153 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'107, mlcod=14'105)", 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 25.320303095, 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046950569999999999, 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.533 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141083+0000", 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:36.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141104+0000", 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0945999999999999e-05 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141127+0000", 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3054e-05 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141149+0000", 2024-09-13T05:18:36.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1690000000000001e-05 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141266+0000", 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000117359 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141303+0000", 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7013e-05 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145722+0000", 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044190260000000004 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145780+0000", 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7751999999999999e-05 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 25.319094983999999, 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:18:36.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.541 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:156 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'110, mlcod=14'105)", 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 25.318516112000001, 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047886200000000004, 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142871+0000", 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142890+0000", 2024-09-13T05:18:36.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9882999999999999e-05 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142944+0000", 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3465999999999999e-05 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142959+0000", 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4644000000000001e-05 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144037+0000", 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001077988 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144094+0000", 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7079e-05 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147601+0000", 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035074400000000001 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147660+0000", 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9243000000000002e-05 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:159 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'113, mlcod=14'105)", 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 25.316831012000002, 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044745929999999998, 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:36.546 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144556+0000", 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:36.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144583+0000", 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7416e-05 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144632+0000", 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9453999999999998e-05 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144651+0000", 2024-09-13T05:18:36.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8635999999999999e-05 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.146970+0000", 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023192159999999998 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147003+0000", 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2595999999999998e-05 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:36.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.148955+0000", 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001951595 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.149031+0000", 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6922999999999995e-05 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:36.550 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:36.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:36.552 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:36.547+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:36.653 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:36.647+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:36.672 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:36.672 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:36.672 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:36.672 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:36.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:36.773 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:36.771+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:36.847 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:36.848+0000 7f7db577d7c0 -1 Falling back to public interface 2024-09-13T05:18:36.849 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:36.849 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:36.849 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:36.849 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:36.850 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:36.854 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:36.851+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:36.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:36.954 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:36.951+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:37.027 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.890275568, 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.028 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:18:37.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.889635599000002, 2024-09-13T05:18:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:18:37.036 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.894762537999998, 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:37.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:18:37.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.888650746, 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:18:37.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:18:37.046 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.884981008, 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:37.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:18:37.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:18:37.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:18:37.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.878843288999999, 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.057 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:37.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:37.059 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:37.060+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:37.159 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:37.160+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:37.219 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:18:37.230 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:37.230 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:37.231 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:37.231 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:37.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:37.289 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:37.287+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:37.361 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:37.360+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:37.408 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:37.408 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:37.408 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:37.408 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:37.408 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:37.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:37.469 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:37.467+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:37.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:18:37.549 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.688354156999999, 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010082459, 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858216+0000", 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858229+0000", 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.305e-05 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858266+0000", 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7523e-05 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858283+0000", 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6597000000000001e-05 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865356+0000", 2024-09-13T05:18:37.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070732859999999998 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865490+0000", 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133442 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865698+0000", 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208202 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867616+0000", 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019181409999999999 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868169+0000", 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055310299999999999 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868246+0000", 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6866000000000006e-05 2024-09-13T05:18:37.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868300+0000", 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3912999999999997e-05 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:37.556 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.688265950999998, 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010336577, 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:37.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858304+0000", 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858318+0000", 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3963e-05 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858337+0000", 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8482999999999999e-05 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858351+0000", 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4195999999999999e-05 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866611+0000", 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082601199999999993 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866763+0000", 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015145600000000001 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866987+0000", 2024-09-13T05:18:37.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224441 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867784+0000", 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079658099999999998 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868531+0000", 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074704700000000003 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868596+0000", 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4966000000000001e-05 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868642+0000", 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6607000000000003e-05 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.562 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.594367388999999, 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:18:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.568 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.594160231, 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:18:37.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:37.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:18:37.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:18:37.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.416921502000001, 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010071422, 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 107 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129649+0000", 2024-09-13T05:18:37.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129656+0000", 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129668+0000", 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2463000000000001e-05 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129675+0000", 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1999999999999997e-06 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131247+0000", 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015719359999999999 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131321+0000", 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4232000000000003e-05 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131413+0000", 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1525000000000002e-05 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.133468+0000", 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002055177 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139645+0000", 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061772490000000001 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139691+0000", 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5967000000000002e-05 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139722+0000", 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0273e-05 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.580 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.416177276999999, 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:18:37.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.587 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.587 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:37.587 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.416048405000002, 2024-09-13T05:18:37.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:18:37.587 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.587 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:37.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:37.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:18:37.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.415763113000001, 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010554714999999999, 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 115 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.594 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130808+0000", 2024-09-13T05:18:37.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130882+0000", 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4449999999999994e-05 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130891+0000", 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7190000000000007e-06 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130897+0000", 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135420+0000", 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045232060000000001 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135472+0000", 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1530999999999998e-05 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135557+0000", 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5192999999999996e-05 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136761+0000", 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001204072 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141283+0000", 2024-09-13T05:18:37.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045216099999999997 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141331+0000", 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8094999999999999e-05 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141363+0000", 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2319e-05 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.599 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.415668845999999, 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010559903000000001, 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 116 2024-09-13T05:18:37.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130902+0000", 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130905+0000", 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130909+0000", 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7239999999999998e-06 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130914+0000", 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1800000000000004e-06 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135956+0000", 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050418850000000003 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136008+0000", 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1694999999999999e-05 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136090+0000", 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2527000000000005e-05 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137204+0000", 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011136640000000001 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141406+0000", 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042020039999999996 2024-09-13T05:18:37.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141438+0000", 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2277999999999998e-05 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141463+0000", 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4396000000000002e-05 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.415650991, 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:18:37.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:18:37.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:18:37.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:18:37.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:18:37.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.611 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.031270668000001, 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:18:37.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.617 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.031233532000002, 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:18:37.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:18:37.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.623 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.031127612999999, 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:18:37.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:37.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:18:37.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:18:37.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.030855588000001, 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.630 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:18:37.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:18:37.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.030812002000001, 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.636 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:18:37.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:18:37.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:37.642 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.030779774999999, 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:18:37.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:37.648 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.030568627000001, 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:18:37.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:18:37.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:37.654 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.030532334, 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:18:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:37.660 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.030493883999998, 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:37.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:18:37.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:37.666 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.030318982000001, 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:37.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:37.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:18:37.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:37.672 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:37.673 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:37.674 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:37.675 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:37.675 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:37.676 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:37.671+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:37.690 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:37.691 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:37.691 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-13T05:18:37.711 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:37.711 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:37.732 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:37.733 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:37.736 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:18:37.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:18:37.775 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:37.776+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:37.805 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:37.805 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:37.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:37.876 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:37.871+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:37.945 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:37.972 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:37.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:38.018 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:38.015+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:38.078 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:38.075+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:38.148 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:38.148 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:38.163 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:38.205 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:38.203+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:38.280 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:38.275+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:38.294 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:38.294 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:38.308 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:38.381 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:38.380+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:38.428 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:38.428 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:38.442 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:38.482 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:38.479+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:38.583 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:38.583+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:38.613 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:38.613 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:38.627 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:38.684 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:38.683+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:38.745 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:38.745 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:38.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:38.785 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:38.783+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:38.876 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:38.876 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:38.886 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:38.883+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:38.890 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:38.988 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:38.983+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:39.004 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:39.004 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:39.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:39.089 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:39.088+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:39.157 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:39.157 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:39.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:39.217 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:39.215+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:39.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:18:39.291 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:39.292+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:39.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:39.391 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:39.387+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:39.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:39.529 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:39.527+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:39.593 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:39.592+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:39.613 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:39.628 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:39.695 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:39.696+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:39.737 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:39.795 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:39.796+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:39.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:39.874 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:39.892 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:39.896 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:39.896+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:39.908 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:39.908 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-13T05:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:39.928 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.074120835999999, 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010082459, 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858216+0000", 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858229+0000", 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.305e-05 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858266+0000", 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7523e-05 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858283+0000", 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6597000000000001e-05 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865356+0000", 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070732859999999998 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865490+0000", 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133442 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865698+0000", 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208202 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867616+0000", 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019181409999999999 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868169+0000", 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055310299999999999 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868246+0000", 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6866000000000006e-05 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868300+0000", 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3912999999999997e-05 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.074032630000001, 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010336577, 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858304+0000", 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858318+0000", 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3963e-05 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858337+0000", 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8482999999999999e-05 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858351+0000", 2024-09-13T05:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4195999999999999e-05 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866611+0000", 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082601199999999993 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866763+0000", 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015145600000000001 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866987+0000", 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224441 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867784+0000", 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079658099999999998 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868531+0000", 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074704700000000003 2024-09-13T05:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868596+0000", 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4966000000000001e-05 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868642+0000", 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6607000000000003e-05 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.980134068000002, 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.97992691, 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.802688181000001, 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010071422, 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 107 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129649+0000", 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129656+0000", 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129668+0000", 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2463000000000001e-05 2024-09-13T05:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129675+0000", 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1999999999999997e-06 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131247+0000", 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015719359999999999 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131321+0000", 2024-09-13T05:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4232000000000003e-05 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131413+0000", 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1525000000000002e-05 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.133468+0000", 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002055177 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139645+0000", 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061772490000000001 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139691+0000", 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5967000000000002e-05 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139722+0000", 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0273e-05 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.801943955999999, 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.801815084000001, 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.801529792, 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010554714999999999, 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 115 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130808+0000", 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130882+0000", 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4449999999999994e-05 2024-09-13T05:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130891+0000", 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7190000000000007e-06 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130897+0000", 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135420+0000", 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045232060000000001 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135472+0000", 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1530999999999998e-05 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135557+0000", 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5192999999999996e-05 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136761+0000", 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001204072 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141283+0000", 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045216099999999997 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141331+0000", 2024-09-13T05:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8094999999999999e-05 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141363+0000", 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2319e-05 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.801435524999999, 2024-09-13T05:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010559903000000001, 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 116 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130902+0000", 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130905+0000", 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130909+0000", 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7239999999999998e-06 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130914+0000", 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1800000000000004e-06 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135956+0000", 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050418850000000003 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136008+0000", 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1694999999999999e-05 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136090+0000", 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2527000000000005e-05 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137204+0000", 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011136640000000001 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141406+0000", 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042020039999999996 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141438+0000", 2024-09-13T05:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2277999999999998e-05 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141463+0000", 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4396000000000002e-05 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.801417669999999, 2024-09-13T05:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.417037347000001, 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.417000211000001, 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.416894291999999, 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.416622267000001, 2024-09-13T05:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.416578681000001, 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.416546453999999, 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.416335306000001, 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.416299013, 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.416260563000002, 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.416085661, 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:40.058 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:40.059 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:18:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:40.059 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:18:40.059 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:40.055+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:40.158 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:40.160+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:18:40.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:40.261 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:40.259+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:40.360 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:40.360+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:40.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:40.461 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:40.464+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.381802176000001, 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:40.528 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:40.562 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:40.559+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:40.624 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:40.624+0000 7f7db577d7c0 -1 osd.3 20 log_to_monitors true 2024-09-13T05:18:40.664 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:40.663+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:40.667 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:40.668 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:40.668 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:40.668 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:40.681 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:40.765 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:40.764+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:40.798 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:40.798 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:40.798 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:40.798 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:40.798 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:40.812 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:40.902 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:40.904+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:40.966 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:40.968+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:40.976 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-13T05:18:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 23.839558823000001, 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001669217, 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139957+0000", 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139968+0000", 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.007e-05 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139993+0000", 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5945999999999999e-05 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140008+0000", 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4926000000000001e-05 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140092+0000", 2024-09-13T05:18:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3282999999999995e-05 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140137+0000", 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5195999999999999e-05 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141553+0000", 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014160659999999999 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141627+0000", 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4330000000000002e-05 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 22.829854628, 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001729281, 2024-09-13T05:18:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149679+0000", 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7090000000000001e-05 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149706+0000", 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7192999999999999e-05 2024-09-13T05:18:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149721+0000", 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.45e-05 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149824+0000", 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000102959 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149862+0000", 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8667000000000001e-05 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151293+0000", 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001431332 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151392+0000", 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8023000000000007e-05 2024-09-13T05:18:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:40.986 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:40.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:41.067 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:41.063+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:41.158 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:41.158 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:41.158 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:41.158 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:41.168 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:41.163+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:41.173 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:41.269 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:41.263+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:41.289 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:41.289 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:41.289 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:41.289 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:41.289 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:41.290 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:41.292+0000 7f7da9544640 -1 osd.3 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T05:18:41.304 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:41.362 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:18:41.370 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:41.367+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:41.426 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:41.426 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:41.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:41.426 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:41.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.426 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'53)", 2024-09-13T05:18:41.426 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:41.426 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.466267555000002, 2024-09-13T05:18:41.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050448339999999998, 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961433+0000", 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961492+0000", 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9488999999999999e-05 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961517+0000", 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5296999999999999e-05 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961529+0000", 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1763000000000001e-05 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961621+0000", 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1636000000000002e-05 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961656+0000", 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5363e-05 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966359+0000", 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047030839999999997 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966479+0000", 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119839 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.430 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.295175798999999, 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:18:41.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:18:41.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:18:41.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.29454454, 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:18:41.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:18:41.439 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.293899418999999, 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:18:41.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:18:41.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.293392055999998, 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:18:41.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.447 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.292866620000002, 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:18:41.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:18:41.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:18:41.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.292229108000001, 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:18:41.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:41.457 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.2916524, 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:18:41.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:18:41.461 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.291100926999999, 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.462 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:18:41.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:18:41.466 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.290640344, 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:18:41.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:18:41.470 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.290130458, 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:18:41.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:18:41.475 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.289703395, 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:41.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:18:41.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:116 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'78, mlcod=14'65)", 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.289242708, 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050255040000000001, 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.480 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138458+0000", 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138469+0000", 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1596e-05 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138490+0000", 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0446e-05 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138521+0000", 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0843000000000003e-05 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142091+0000", 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035703929999999998 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142146+0000", 2024-09-13T05:18:41.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4873000000000001e-05 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143395+0000", 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012491169999999999 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143485+0000", 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9419000000000006e-05 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.484 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:117 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'79, mlcod=14'65)", 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.288750999000001, 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050956009999999999, 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.485 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138949+0000", 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:41.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138962+0000", 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2876e-05 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138987+0000", 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4652999999999999e-05 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139021+0000", 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4122999999999998e-05 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142737+0000", 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003715472 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142773+0000", 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6086e-05 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144004+0000", 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001231631 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144046+0000", 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1996000000000002e-05 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:132 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'90, mlcod=14'86)", 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 30.905563733000001, 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005226835, 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522137+0000", 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522142+0000", 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7040000000000002e-06 2024-09-13T05:18:41.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522153+0000", 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0573e-05 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522158+0000", 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9389999999999996e-06 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522191+0000", 2024-09-13T05:18:41.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3571000000000001e-05 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522204+0000", 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3169999999999999e-05 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527326+0000", 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00512153 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527365+0000", 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8971999999999999e-05 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 30.902943677, 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.494 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:18:41.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.498 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:153 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'107, mlcod=14'105)", 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 30.286616976000001, 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046950569999999999, 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141083+0000", 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141104+0000", 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0945999999999999e-05 2024-09-13T05:18:41.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141127+0000", 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3054e-05 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141149+0000", 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1690000000000001e-05 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141266+0000", 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000117359 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141303+0000", 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7013e-05 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145722+0000", 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044190260000000004 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145780+0000", 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7751999999999999e-05 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 30.285408865000001, 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.503 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:18:41.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:18:41.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:156 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'110, mlcod=14'105)", 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 30.284829992999999, 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047886200000000004, 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142871+0000", 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142890+0000", 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9882999999999999e-05 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142944+0000", 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3465999999999999e-05 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142959+0000", 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4644000000000001e-05 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144037+0000", 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001077988 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144094+0000", 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7079e-05 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147601+0000", 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035074400000000001 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147660+0000", 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9243000000000002e-05 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:159 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'113, mlcod=14'105)", 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 30.283144892999999, 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044745929999999998, 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:41.512 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:41.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144556+0000", 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144583+0000", 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7416e-05 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144632+0000", 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9453999999999998e-05 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144651+0000", 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8635999999999999e-05 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.146970+0000", 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023192159999999998 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147003+0000", 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2595999999999998e-05 2024-09-13T05:18:41.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.148955+0000", 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001951595 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.149031+0000", 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6922999999999995e-05 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:41.516 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:41.517 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:41.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:41.519 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:41.519 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:41.519 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:41.519 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:41.520 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:41.520+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:41.619 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:41.620+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:41.656 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:41.657 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:41.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:41.721 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:41.720+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:41.788 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:41.789 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:41.789 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:41.789 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:41.789 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:41.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:41.822 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:41.824+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:41.922 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:41.919+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:41.930 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 25.793082403, 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:41.931 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:41.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 25.792442434000002, 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:41.935 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:41.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:18:41.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.797569373000002, 2024-09-13T05:18:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:18:41.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:41.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:41.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:41.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:41.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:18:41.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.940 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:41.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:41.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:18:41.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:18:41.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:41.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:41.946 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.791457581, 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.787787843, 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:41.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:18:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:18:41.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:41.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:41.958 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.781650123999999, 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:41.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:41.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:41.962 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:41.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:42.053 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:42.051+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:42.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:18:42.125 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:42.124+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:42.128 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:42.129 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:42.129 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:42.129 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:42.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:42.199 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:42.200 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:42.216 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:42.225 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:42.223+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:42.232 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:42.233 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:42.233 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-13T05:18:42.254 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:42.255 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:42.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:18:42.272 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:42.272 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:42.273 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:18:42.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:18:42.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:42.327 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:42.328+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:42.428 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:42.428+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:42.453 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:42.453 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:42.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:42.528 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:42.523+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:42.580 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:42.580 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:42.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:42.637 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:42.635+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:42.730 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:42.732+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:42.760 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:42.760 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:42.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:42.831 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:42.832+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:42.898 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:42.898 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:42.913 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:42.931 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:42.927+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:43.032 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:43.032+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:43.046 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:43.046 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:43.060 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:43.133 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:43.131+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:43.225 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:43.225 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:43.234 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:43.236+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:43.240 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:18:43.341 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:43.339+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:43.400 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:43.400 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:43.415 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:43.461 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:43.464+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:43.537 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:43.537 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:43.537 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:43.536+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:43.550 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:43.638 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:43.635+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:43.717 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:43.717 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:43.731 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:43.778 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:43.780+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:43.839 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:43.835+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:43.852 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:43.853 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:43.867 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:18:43.954 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:43.956+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:44.029 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:44.029 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:44.029 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:44.029 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:44.041 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:44.040+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:44.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:18:44.143 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:44.139+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:44.205 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:44.206 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:44.206 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:44.206 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:44.206 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:44.221 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:18:44.244 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:44.239+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:44.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:18:44.345 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:44.339+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:44.345 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:44.345 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:44.345 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:44.346 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [] 2024-09-13T05:18:44.346 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:44.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:44.434 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:44.434 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:44.445 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:44.443+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:44.450 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:44.450 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:44.467 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:44.468 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:44.483 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:44.483 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:44.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:18:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T05:18:44.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:44.507 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:44.508 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:44.527 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:44.546 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:44.548+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:44.639 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:44.647 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:44.648+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:44.748 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:44.748+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.627363415000001, 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:18:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.626723446, 2024-09-13T05:18:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:18:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:18:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:18:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.631850385, 2024-09-13T05:18:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:18:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:18:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:18:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.625738593000001, 2024-09-13T05:18:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:18:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:18:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:18:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:18:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:18:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:18:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:18:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.622068854999998, 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:18:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:18:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.615931136, 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:18:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:44.801 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:44.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:44.849 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:44.847+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:44.921 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:44.921 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:44.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:44.977 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:44.975+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:45.049 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:45.047+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:45.104 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:45.105 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:45.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:45.165 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:45.163+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:45.251 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:45.247+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:45.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:18:45.299 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:45.299 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:45.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.437434148999998, 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010082459, 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:18:45.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858216+0000", 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858229+0000", 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.305e-05 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858266+0000", 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7523e-05 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858283+0000", 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6597000000000001e-05 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865356+0000", 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070732859999999998 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865490+0000", 2024-09-13T05:18:45.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133442 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865698+0000", 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208202 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867616+0000", 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019181409999999999 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868169+0000", 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055310299999999999 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868246+0000", 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6866000000000006e-05 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868300+0000", 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3912999999999997e-05 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.437345942999997, 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010336577, 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.306 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858304+0000", 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:45.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858318+0000", 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3963e-05 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858337+0000", 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8482999999999999e-05 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858351+0000", 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4195999999999999e-05 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866611+0000", 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082601199999999993 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866763+0000", 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015145600000000001 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866987+0000", 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224441 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867784+0000", 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079658099999999998 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868531+0000", 2024-09-13T05:18:45.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074704700000000003 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868596+0000", 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4966000000000001e-05 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868642+0000", 2024-09-13T05:18:45.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6607000000000003e-05 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.343447380999997, 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.312 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:18:45.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.343240223000002, 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.318 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:18:45.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.166001494, 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010071422, 2024-09-13T05:18:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 107 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129649+0000", 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129656+0000", 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2024-09-13T05:18:45.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129668+0000", 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2463000000000001e-05 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129675+0000", 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1999999999999997e-06 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131247+0000", 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015719359999999999 2024-09-13T05:18:45.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131321+0000", 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4232000000000003e-05 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131413+0000", 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1525000000000002e-05 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.133468+0000", 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002055177 2024-09-13T05:18:45.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139645+0000", 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061772490000000001 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139691+0000", 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5967000000000002e-05 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139722+0000", 2024-09-13T05:18:45.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0273e-05 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.165257269000001, 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.165128396999997, 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.164843105000003, 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010554714999999999, 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 115 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130808+0000", 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130882+0000", 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4449999999999994e-05 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130891+0000", 2024-09-13T05:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7190000000000007e-06 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130897+0000", 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135420+0000", 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045232060000000001 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135472+0000", 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1530999999999998e-05 2024-09-13T05:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135557+0000", 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5192999999999996e-05 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136761+0000", 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001204072 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141283+0000", 2024-09-13T05:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045216099999999997 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141331+0000", 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8094999999999999e-05 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141363+0000", 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2319e-05 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.164748838000001, 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010559903000000001, 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 116 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130902+0000", 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130905+0000", 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130909+0000", 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7239999999999998e-06 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130914+0000", 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1800000000000004e-06 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135956+0000", 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050418850000000003 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136008+0000", 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1694999999999999e-05 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136090+0000", 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2527000000000005e-05 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137204+0000", 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011136640000000001 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141406+0000", 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042020039999999996 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141438+0000", 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2277999999999998e-05 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141463+0000", 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4396000000000002e-05 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.164730982999998, 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.357 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.780350660000003, 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:18:45.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:18:45.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:45.363 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.780313524, 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:18:45.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.780207605000001, 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:18:45.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:18:45.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:18:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.77993558, 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.375 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:18:45.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:18:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:18:45.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.779891994000003, 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:18:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:18:45.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:18:45.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.779859766999998, 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:18:45.388 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:18:45.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:18:45.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.779648619, 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.394 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:18:45.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:18:45.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.779612325999999, 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.400 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.405 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.779573876000001, 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:18:45.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.411 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.779398974000003, 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:45.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:18:45.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:18:45.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.417 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:45.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:45.419 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:45.420+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:45.520 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:45.515+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:45.584 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:45.584 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:45.584 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:45.584 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:45.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:45.620 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:45.620+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:45.721 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:45.724+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:45.760 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:45.761 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:45.775 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:45.822 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:45.819+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:45.902 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 29.770994542, 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:45.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:45.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:45.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:45.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:45.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:45.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:18:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:18:45.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:45.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:18:45.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:18:45.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.908 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:45.908 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:45.908 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:45.918 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:45.961 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:45.959+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:46.024 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:46.024+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:46.050 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:46.050 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:46.050 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:46.050 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:46.065 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:46.125 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:46.123+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:46.226 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:46.228+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:46.228 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:46.228 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:46.228 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:46.229 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:46.229 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:46.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:18:46.330 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:46.332+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:46.408 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:46.408 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:46.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:46.408 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:46.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.408 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-13T05:18:46.408 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 29.271399933000001, 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001669217, 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:46.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139957+0000", 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139968+0000", 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.007e-05 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139993+0000", 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5945999999999999e-05 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140008+0000", 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4926000000000001e-05 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140092+0000", 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3282999999999995e-05 2024-09-13T05:18:46.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140137+0000", 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5195999999999999e-05 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141553+0000", 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014160659999999999 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141627+0000", 2024-09-13T05:18:46.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4330000000000002e-05 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.261695738, 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001729281, 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:46.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149679+0000", 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7090000000000001e-05 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149706+0000", 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7192999999999999e-05 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149721+0000", 2024-09-13T05:18:46.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.45e-05 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149824+0000", 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000102959 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149862+0000", 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8667000000000001e-05 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151293+0000", 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001431332 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151392+0000", 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8023000000000007e-05 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:46.417 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:46.422 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:46.466 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:46.468+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:46.520 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 3] 2024-09-13T05:18:46.521 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:18:46.521 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T05:18:46.521 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.980807 2024-09-13T05:18:46.521 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.9808065074027751 2024-09-13T05:18:46.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:18:46.565 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:46.563+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:46.588 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:46.588 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:46.588 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:46.588 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:46.601 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:46.629 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:46.627+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:46.719 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:46.719 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:46.719 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:46.720 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:46.720 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:46.726 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:46.730 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:46.732+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:46.733 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:46.744 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:46.763 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:46.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:18:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:18:46.799 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:46.799 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:46.816 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:46.816 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:46.831 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:46.827+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:46.902 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:46.902 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:46.917 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:18:46.932 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:46.927+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:47.032 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:47.027+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:47.039 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:47.039 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:47.053 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:18:47.133 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:47.131+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:47.175 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:47.175 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:47.189 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:18:47.234 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:47.231+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:47.281 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.3 primary-affinity to 0.980807 (8642782) 2024-09-13T05:18:47.312 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:47.312 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:47.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:47.345 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:47.343+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:47.436 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:47.436+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:47.475 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:47.475 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:47.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:47.537 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:47.540+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:47.616 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:47.616 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:47.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:47.677 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:47.675+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:47.739 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:47.735+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:47.759 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:47.759 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:47.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:47.861 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:47.859+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:47.941 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:47.939+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:47.942 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:47.942 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:47.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:48.045 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:48.048+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:48.126 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:48.126 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:48.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:48.145 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:48.148+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:48.247 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:48.248+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:48.266 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:48.267 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:48.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:48.304 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:18:48.369 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:48.367+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:48.407 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:48.407 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:48.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:48.448 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:48.448+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:48.549 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:48.543+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:48.580 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:48.580 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:48.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:48.650 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:48.652+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:48.751 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:48.747+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:48.769 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:48.769 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:48.784 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:48.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:18:48.851 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:48.852+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:48.913 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:48.913 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:48.928 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:48.970 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:48.972+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:48.982 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:48.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:48.998 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:49.015 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:49.016 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:49.033 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:18:49.033 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:18:49.050 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:49.051 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:49.053 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:49.052+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:49.070 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:49.070 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:49.089 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:49.089 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:49.089 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:49.089 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:49.089 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:49.104 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:49.190 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:49.192+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:49.254 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:49.256+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 32.126474346999998, 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001669217, 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.263 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139957+0000", 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139968+0000", 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.007e-05 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139993+0000", 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5945999999999999e-05 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140008+0000", 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4926000000000001e-05 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140092+0000", 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3282999999999995e-05 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140137+0000", 2024-09-13T05:18:49.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5195999999999999e-05 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141553+0000", 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014160659999999999 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141627+0000", 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4330000000000002e-05 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.267 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.116770152000001, 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001729281, 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149679+0000", 2024-09-13T05:18:49.269 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7090000000000001e-05 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149706+0000", 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7192999999999999e-05 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149721+0000", 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.45e-05 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149824+0000", 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000102959 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149862+0000", 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8667000000000001e-05 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151293+0000", 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001431332 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.272 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151392+0000", 2024-09-13T05:18:49.272 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8023000000000007e-05 2024-09-13T05:18:49.272 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.272 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.272 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.272 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.272 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.272 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:49.276 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:49.292 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:18:49.361 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:49.364+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:49.423 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:49.423 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:49.423 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:49.423 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:49.436 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:49.477 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:49.480+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:49.557 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:49.560+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:49.583 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:49.583 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:49.583 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:49.583 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:49.584 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:49.596 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:49.658 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:49.655+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:49.745 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:49.745 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:49.745 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'53)", 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.784661919999998, 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050448339999999998, 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.746 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961433+0000", 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.747 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961492+0000", 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9488999999999999e-05 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961517+0000", 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5296999999999999e-05 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961529+0000", 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1763000000000001e-05 2024-09-13T05:18:49.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961621+0000", 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1636000000000002e-05 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961656+0000", 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5363e-05 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966359+0000", 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047030839999999997 2024-09-13T05:18:49.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966479+0000", 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119839 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.613570164000002, 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.750 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:18:49.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.612938905, 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:49.755 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.758 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:18:49.759 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.612293784000002, 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:18:49.763 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.611786420999998, 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.764 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:18:49.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:18:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:18:49.768 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.611260985000001, 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:18:49.770 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:18:49.771 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.772 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.610623472999997, 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.773 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:18:49.774 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:18:49.775 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.776 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.610046765, 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.777 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.778 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.779 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.780 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.781 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.609495291999998, 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.782 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:18:49.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.784 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.785 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.609034708999999, 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.786 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:18:49.787 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.788 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.789 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.790 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.608524823000003, 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.791 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.792 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:18:49.793 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:18:49.794 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.60809776, 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.795 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.796 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:18:49.797 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:18:49.798 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:116 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'78, mlcod=14'65)", 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.607637072999999, 2024-09-13T05:18:49.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050255040000000001, 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138458+0000", 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138469+0000", 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1596e-05 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.801 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138490+0000", 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0446e-05 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138521+0000", 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0843000000000003e-05 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142091+0000", 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035703929999999998 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.802 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142146+0000", 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4873000000000001e-05 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143395+0000", 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012491169999999999 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143485+0000", 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9419000000000006e-05 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.803 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:117 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'79, mlcod=14'65)", 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.607145363999997, 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050956009999999999, 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138949+0000", 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138962+0000", 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2876e-05 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138987+0000", 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4652999999999999e-05 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139021+0000", 2024-09-13T05:18:49.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4122999999999998e-05 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142737+0000", 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003715472 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142773+0000", 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6086e-05 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144004+0000", 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001231631 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144046+0000", 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1996000000000002e-05 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.808 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:132 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'90, mlcod=14'86)", 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.223958097999997, 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005226835, 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:49.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522137+0000", 2024-09-13T05:18:49.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:49.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522142+0000", 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7040000000000002e-06 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522153+0000", 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0573e-05 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522158+0000", 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9389999999999996e-06 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522191+0000", 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3571000000000001e-05 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522204+0000", 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3169999999999999e-05 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527326+0000", 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00512153 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527365+0000", 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8971999999999999e-05 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:18:49.813 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 39.221338041999999, 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:49.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:18:49.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:153 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'107, mlcod=14'105)", 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.605011341000001, 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046950569999999999, 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141083+0000", 2024-09-13T05:18:49.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141104+0000", 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0945999999999999e-05 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141127+0000", 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3054e-05 2024-09-13T05:18:49.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141149+0000", 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1690000000000001e-05 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141266+0000", 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000117359 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141303+0000", 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7013e-05 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145722+0000", 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044190260000000004 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145780+0000", 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7751999999999999e-05 2024-09-13T05:18:49.822 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.603803229999997, 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.825 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.827 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:156 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'110, mlcod=14'105)", 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.603224357999999, 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047886200000000004, 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142871+0000", 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142890+0000", 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9882999999999999e-05 2024-09-13T05:18:49.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142944+0000", 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3465999999999999e-05 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142959+0000", 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4644000000000001e-05 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144037+0000", 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001077988 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144094+0000", 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7079e-05 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147601+0000", 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035074400000000001 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147660+0000", 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9243000000000002e-05 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:159 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'113, mlcod=14'105)", 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.601539258000003, 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044745929999999998, 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:49.832 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:49.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144556+0000", 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144583+0000", 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7416e-05 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144632+0000", 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9453999999999998e-05 2024-09-13T05:18:49.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144651+0000", 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8635999999999999e-05 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.146970+0000", 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023192159999999998 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147003+0000", 2024-09-13T05:18:49.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2595999999999998e-05 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.148955+0000", 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001951595 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.149031+0000", 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6922999999999995e-05 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.836 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.837 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:49.837 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:49.837 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:49.839 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:18:49.839 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:49.840+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:49.939 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:49.940+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:50.003 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:50.004 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:50.004 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:50.004 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:50.018 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:18:50.041 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:50.035+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:50.141 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:50.144+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:50.182 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:50.182 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:50.183 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:50.183 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:50.183 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:50.196 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:18:50.242 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:50.239+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:50.343 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:50.344+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:50.358 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:50.358 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:50.358 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:50.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [] 2024-09-13T05:18:50.359 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:50.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:50.444 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:50.439+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:50.500 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:50.501 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:50.501 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:50.501 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:50.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:50.545 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:50.548+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:50.646 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:50.648+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:50.675 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:50.675 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:50.675 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:50.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:50.747 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:50.743+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:50.818 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.679951160000002, 2024-09-13T05:18:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:18:50.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:50.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.679311190999996, 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:18:50.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:18:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.684438129999997, 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:50.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:18:50.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:18:50.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:50.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.678326337999998, 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:18:50.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:18:50.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:50.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:18:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.674656599999999, 2024-09-13T05:18:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:18:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:50.838 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:18:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:50.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:50.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:18:50.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:18:50.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.668518880999997, 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:50.845 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:18:50.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:18:50.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:50.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:18:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:18:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:50.849 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:50.849 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:50.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:50.851 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:50.852+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:50.951 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:50.952+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:51.022 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:51.022 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:51.022 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:51.023 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:51.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:51.052 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:51.052+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:51.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:18:51.153 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:51.151+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:51.155 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:51.156 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:51.171 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:51.227 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:51.228 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:51.244 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:51.253 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:51.251+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:51.261 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:51.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:51.282 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:18:51.282 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:18:51.302 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:51.302 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:51.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:18:51.323 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:51.323 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:51.332 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:51.333 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:51.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:51.354 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:51.356+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:51.455 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:51.451+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:51.518 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:51.519 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:51.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:51.557 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:51.556+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:51.657 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:51.660+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:51.705 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:51.705 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:51.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:51.765 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:51.763+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:51.840 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:51.840 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:51.853 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:51.897 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:51.895+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:51.960 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:51.960+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:51.985 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:51.985 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:51.999 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:52.085 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:52.088+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:52.158 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:52.159 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:52.162 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:52.164+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:52.173 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:52.263 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:52.264+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:52.304 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 3] 2024-09-13T05:18:52.305 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:18:52.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:18:52.313 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:18:52.344 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:52.344 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:52.358 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:52.385 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:52.383+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:52.465 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:52.459+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:52.520 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:52.520 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:52.534 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:52.566 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:52.563+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:52.606 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:18:52.625 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-09-13T05:18:52.313562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":178588,"kb_used_data":3076,"kb_used_omap":12,"kb_used_meta":175411,"kb_avail":566052452,"statfs":{"total":579820584960,"available":579637710848,"internally_reserved":0,"allocated":3149824,"data_stored":1867844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12399,"internal_metadata":179621777},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.033746"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-13T05:18:43.161305+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:18:43.161305+0000","last_peered":"2024-09-13T05:18:43.161305+0000","last_clean":"2024-09-13T05:18:43.161305+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:18:43.161305+0000","last_undegraded":"2024-09-13T05:18:43.161305+0000","last_fullsized":"2024-09-13T05:18:43.161305+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-13T05:18:43.238584+0000","last_change":"2024-09-13T05:18:18.143477+0000","last_active":"2024-09-13T05:18:43.238584+0000","last_peered":"2024-09-13T05:18:43.238584+0000","last_clean":"2024-09-13T05:18:43.238584+0000","last_became_active":"2024-09-13T05:18:16.129462+0000","last_became_peered":"2024-09-13T05:18:16.129462+0000","last_unstale":"2024-09-13T05:18:43.238584+0000","last_undegraded":"2024-09-13T05:18:43.238584+0000","last_fullsized":"2024-09-13T05:18:43.238584+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:33:02.115421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076334099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-13T05:18:43.161224+0000","last_change":"2024-09-13T05:18:18.324354+0000","last_active":"2024-09-13T05:18:43.161224+0000","last_peered":"2024-09-13T05:18:43.161224+0000","last_clean":"2024-09-13T05:18:43.161224+0000","last_became_active":"2024-09-13T05:18:16.125619+0000","last_became_peered":"2024-09-13T05:18:16.125619+0000","last_unstale":"2024-09-13T05:18:43.161224+0000","last_undegraded":"2024-09-13T05:18:43.161224+0000","last_fullsized":"2024-09-13T05:18:43.161224+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:40.629120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-13T05:18:43.280926+0000","last_change":"2024-09-13T05:18:43.275214+0000","last_active":"2024-09-13T05:18:43.280926+0000","last_peered":"2024-09-13T05:18:43.280926+0000","last_clean":"2024-09-13T05:18:43.280926+0000","last_became_active":"2024-09-13T05:18:43.274886+0000","last_became_peered":"2024-09-13T05:18:43.274886+0000","last_unstale":"2024-09-13T05:18:43.280926+0000","last_undegraded":"2024-09-13T05:18:43.280926+0000","last_fullsized":"2024-09-13T05:18:43.280926+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:53.129994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-13T05:18:43.417617+0000","last_change":"2024-09-13T05:18:18.147989+0000","last_active":"2024-09-13T05:18:43.417617+0000","last_peered":"2024-09-13T05:18:43.417617+0000","last_clean":"2024-09-13T05:18:43.417617+0000","last_became_active":"2024-09-13T05:18:16.129277+0000","last_became_peered":"2024-09-13T05:18:16.129277+0000","last_unstale":"2024-09-13T05:18:43.417617+0000","last_undegraded":"2024-09-13T05:18:43.417617+0000","last_fullsized":"2024-09-13T05:18:43.417617+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:59:39.922642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-13T05:18:47.284495+0000","last_change":"2024-09-13T05:18:43.272711+0000","last_active":"2024-09-13T05:18:47.284495+0000","last_peered":"2024-09-13T05:18:47.284495+0000","last_clean":"2024-09-13T05:18:47.284495+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:18:47.284495+0000","last_undegraded":"2024-09-13T05:18:47.284495+0000","last_fullsized":"2024-09-13T05:18:47.284495+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:27.589949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-13T05:18:43.269475+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:18:43.269475+0000","last_peered":"2024-09-13T05:18:43.269475+0000","last_clean":"2024-09-13T05:18:43.269475+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:18:43.269475+0000","last_undegraded":"2024-09-13T05:18:43.269475+0000","last_fullsized":"2024-09-13T05:18:43.269475+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-13T05:18:43.161227+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:18:43.161227+0000","last_peered":"2024-09-13T05:18:43.161227+0000","last_clean":"2024-09-13T05:18:43.161227+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:18:43.161227+0000","last_undegraded":"2024-09-13T05:18:43.161227+0000","last_fullsized":"2024-09-13T05:18:43.161227+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":175,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-13T05:18:43.161280+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:18:43.161280+0000","last_peered":"2024-09-13T05:18:43.161280+0000","last_clean":"2024-09-13T05:18:43.161280+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:18:43.161280+0000","last_undegraded":"2024-09-13T05:18:43.161280+0000","last_fullsized":"2024-09-13T05:18:43.161280+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607565,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":921600,"data_stored":705628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":4,"up_from":12,"seq":51539607565,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":319488,"data_stored":110733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28456,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94343384,"statfs":{"total":96636764160,"available":96607625216,"internally_reserved":0,"allocated":352256,"data_stored":124408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]}]},{"osd":2,"up_from":12,"seq":51539607564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":315392,"data_stored":110714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]}]},{"osd":1,"up_from":12,"seq":51539607564,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":909312,"data_stored":701082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.097}]}]},{"osd":0,"up_from":12,"seq":51539607564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":331776,"data_stored":115279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:18:52.651 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:52.651 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:52.666 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:52.669 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:52.667+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:52.768 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:52.767+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:52.786 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:52.786 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:52.800 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:18:52.869 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:52.867+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:52.922 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:52.922 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:52.936 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:18:52.981 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:52.984+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:53.057 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:53.057 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:53.071 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:18:53.071 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:53.072+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:53.171 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:53.172+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:53.237 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:53.238 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:53.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:53.298 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:53.295+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:53.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:18:53.373 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:53.371+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:53.389 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:53.389 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:53.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:53.475 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:53.476+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:53.493 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:53.494 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:53.511 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:53.512 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:53.527 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:53.548 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:18:53.548 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:18:53.568 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:53.568 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:53.569 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:53.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:53.576+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:53.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:53.589 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:53.676 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:53.676+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.581367327999999, 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:18:53.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:18:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:18:53.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.580727359000001, 2024-09-13T05:18:53.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:18:53.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:18:53.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.585854298000001, 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:53.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:18:53.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:53.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.579742506000002, 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:18:53.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:53.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.576072768000003, 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:18:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:53.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:18:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.569935049000001, 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:18:53.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:18:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:53.749 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:53.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:53.777 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:53.776+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:53.877 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:53.878 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:53.878 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:53.880+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:53.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:53.980 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:53.980+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:54.019 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:54.019 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:54.019 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:54.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:54.081 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:54.079+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:54.154 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.295294181000003, 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010082459, 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:18:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858216+0000", 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858229+0000", 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.305e-05 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858266+0000", 2024-09-13T05:18:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7523e-05 2024-09-13T05:18:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858283+0000", 2024-09-13T05:18:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6597000000000001e-05 2024-09-13T05:18:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865356+0000", 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070732859999999998 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865490+0000", 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133442 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865698+0000", 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208202 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867616+0000", 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019181409999999999 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868169+0000", 2024-09-13T05:18:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055310299999999999 2024-09-13T05:18:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868246+0000", 2024-09-13T05:18:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6866000000000006e-05 2024-09-13T05:18:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868300+0000", 2024-09-13T05:18:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3912999999999997e-05 2024-09-13T05:18:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.295205975000002, 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010336577, 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858304+0000", 2024-09-13T05:18:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858318+0000", 2024-09-13T05:18:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3963e-05 2024-09-13T05:18:54.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858337+0000", 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8482999999999999e-05 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858351+0000", 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4195999999999999e-05 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866611+0000", 2024-09-13T05:18:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082601199999999993 2024-09-13T05:18:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866763+0000", 2024-09-13T05:18:54.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015145600000000001 2024-09-13T05:18:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866987+0000", 2024-09-13T05:18:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224441 2024-09-13T05:18:54.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867784+0000", 2024-09-13T05:18:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079658099999999998 2024-09-13T05:18:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868531+0000", 2024-09-13T05:18:54.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074704700000000003 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868596+0000", 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4966000000000001e-05 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868642+0000", 2024-09-13T05:18:54.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6607000000000003e-05 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.201307413000002, 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:18:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:18:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:18:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:18:54.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:18:54.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.201100255, 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:18:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:18:54.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:18:54.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:18:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:18:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:18:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:18:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:18:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:18:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:18:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:18:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:18:54.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:18:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:18:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:18:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:18:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.023861525999997, 2024-09-13T05:18:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010071422, 2024-09-13T05:18:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 107 2024-09-13T05:18:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:18:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129649+0000", 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129656+0000", 2024-09-13T05:18:54.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129668+0000", 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2463000000000001e-05 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129675+0000", 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1999999999999997e-06 2024-09-13T05:18:54.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131247+0000", 2024-09-13T05:18:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015719359999999999 2024-09-13T05:18:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131321+0000", 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4232000000000003e-05 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131413+0000", 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1525000000000002e-05 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.133468+0000", 2024-09-13T05:18:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002055177 2024-09-13T05:18:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139645+0000", 2024-09-13T05:18:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061772490000000001 2024-09-13T05:18:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139691+0000", 2024-09-13T05:18:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5967000000000002e-05 2024-09-13T05:18:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139722+0000", 2024-09-13T05:18:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0273e-05 2024-09-13T05:18:54.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.198 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.023117300999999, 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:18:54.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:18:54.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:18:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:18:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:18:54.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:18:54.205 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.022988429000002, 2024-09-13T05:18:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:18:54.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:18:54.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:18:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:18:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:18:54.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:18:54.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:18:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:18:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:18:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.022703137000001, 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010554714999999999, 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 115 2024-09-13T05:18:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:18:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130808+0000", 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130882+0000", 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4449999999999994e-05 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130891+0000", 2024-09-13T05:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7190000000000007e-06 2024-09-13T05:18:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130897+0000", 2024-09-13T05:18:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:18:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135420+0000", 2024-09-13T05:18:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045232060000000001 2024-09-13T05:18:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135472+0000", 2024-09-13T05:18:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1530999999999998e-05 2024-09-13T05:18:54.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135557+0000", 2024-09-13T05:18:54.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5192999999999996e-05 2024-09-13T05:18:54.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136761+0000", 2024-09-13T05:18:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001204072 2024-09-13T05:18:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141283+0000", 2024-09-13T05:18:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045216099999999997 2024-09-13T05:18:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141331+0000", 2024-09-13T05:18:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8094999999999999e-05 2024-09-13T05:18:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141363+0000", 2024-09-13T05:18:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2319e-05 2024-09-13T05:18:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.224 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.022608869999999, 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010559903000000001, 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 116 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:18:54.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130902+0000", 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130905+0000", 2024-09-13T05:18:54.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130909+0000", 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7239999999999998e-06 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130914+0000", 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1800000000000004e-06 2024-09-13T05:18:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135956+0000", 2024-09-13T05:18:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050418850000000003 2024-09-13T05:18:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136008+0000", 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1694999999999999e-05 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136090+0000", 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2527000000000005e-05 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137204+0000", 2024-09-13T05:18:54.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011136640000000001 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141406+0000", 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042020039999999996 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141438+0000", 2024-09-13T05:18:54.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2277999999999998e-05 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141463+0000", 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4396000000000002e-05 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.022591015000003, 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:18:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:18:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:18:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:18:54.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:18:54.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:18:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:18:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:18:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:54.240 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.638210692000001, 2024-09-13T05:18:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:18:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:18:54.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:18:54.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:18:54.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:18:54.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:18:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:18:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:18:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:18:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:18:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:18:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:18:54.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:18:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:18:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:18:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.638173555999998, 2024-09-13T05:18:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:18:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:18:54.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:18:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.255 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.638067636999999, 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:18:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:18:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:18:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:18:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:18:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:18:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:18:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:18:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:18:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:18:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:18:54.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:18:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.637795611999998, 2024-09-13T05:18:54.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:18:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:18:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.264 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:18:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:18:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:54.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:18:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:18:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:18:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:18:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:18:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:18:54.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:18:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:18:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:18:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:18:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:18:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:18:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:18:54.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.272 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.637752026000001, 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:18:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:18:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:18:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:18:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:18:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:18:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:18:54.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:18:54.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:18:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:18:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:18:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:18:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:18:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:18:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:18:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.280 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.637719799000003, 2024-09-13T05:18:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:18:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:54.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:18:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:18:54.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:18:54.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.637508650999997, 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.288 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:18:54.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:18:54.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.637472357999997, 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.294 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:18:54.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:18:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:18:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:18:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:18:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:18:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:18:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:18:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:18:54.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:18:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:18:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.637433907999998, 2024-09-13T05:18:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:18:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.302 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:18:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:18:54.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:18:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:18:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:18:54.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:18:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:18:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:18:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:54.310 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.637259006000001, 2024-09-13T05:18:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:18:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:18:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:18:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:18:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.311 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:18:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:18:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:18:54.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:18:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:18:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:18:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:18:54.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:18:54.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.318 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:54.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:54.321 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:54.315+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:54.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:18:54.422 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:54.419+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:54.499 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:54.499 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:54.500 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:54.500 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:54.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:54.557 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:54.555+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:54.624 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:54.619+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:54.673 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:54.673 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:54.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:54.729 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:54.727+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:54.826 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:54.823+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:54.850 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.718751453000003, 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:54.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:54.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:18:54.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:54.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:18:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:18:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:54.856 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:54.856 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:54.864 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:54.927 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:54.923+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:54.996 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:54.996 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:54.996 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:54.996 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:55.010 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:55.057 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:55.060+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:55.128 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:55.128+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:55.135 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:55.136 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:55.136 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:55.136 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:55.136 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:55.149 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:55.238 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:55.240+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:55.311 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:55.311 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.174802241000002, 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001669217, 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.312 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139957+0000", 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:55.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139968+0000", 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.007e-05 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139993+0000", 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5945999999999999e-05 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.314 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140008+0000", 2024-09-13T05:18:55.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4926000000000001e-05 2024-09-13T05:18:55.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.315 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140092+0000", 2024-09-13T05:18:55.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3282999999999995e-05 2024-09-13T05:18:55.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140137+0000", 2024-09-13T05:18:55.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5195999999999999e-05 2024-09-13T05:18:55.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141553+0000", 2024-09-13T05:18:55.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014160659999999999 2024-09-13T05:18:55.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.320 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.320 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141627+0000", 2024-09-13T05:18:55.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4330000000000002e-05 2024-09-13T05:18:55.320 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.320 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.320 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.321 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-13T05:18:55.321 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:55.321 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.165098045999997, 2024-09-13T05:18:55.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001729281, 2024-09-13T05:18:55.321 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.322 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.322 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:55.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:55.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:55.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:55.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149679+0000", 2024-09-13T05:18:55.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7090000000000001e-05 2024-09-13T05:18:55.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149706+0000", 2024-09-13T05:18:55.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7192999999999999e-05 2024-09-13T05:18:55.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149721+0000", 2024-09-13T05:18:55.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.45e-05 2024-09-13T05:18:55.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149824+0000", 2024-09-13T05:18:55.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000102959 2024-09-13T05:18:55.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149862+0000", 2024-09-13T05:18:55.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8667000000000001e-05 2024-09-13T05:18:55.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151293+0000", 2024-09-13T05:18:55.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001431332 2024-09-13T05:18:55.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151392+0000", 2024-09-13T05:18:55.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8023000000000007e-05 2024-09-13T05:18:55.330 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.330 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.331 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.331 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.331 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.331 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:55.332 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:18:55.333 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:55.334 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:55.327+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:55.433 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:55.436+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:55.502 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:55.502 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:55.502 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:55.502 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:55.516 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:55.534 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:55.531+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:55.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:18:55.661 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:55.659+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:55.680 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:55.680 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:55.680 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:55.680 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:55.680 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:55.695 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:55.736 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:55.731+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:55.766 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:55.767 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:55.783 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:55.783 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:55.798 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:55.817 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:55.817 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:55.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:55.818 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:55.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.818 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'53)", 2024-09-13T05:18:55.818 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:55.818 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.853554250999998, 2024-09-13T05:18:55.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050448339999999998, 2024-09-13T05:18:55.819 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.819 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.819 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.820 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:55.820 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961433+0000", 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961492+0000", 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9488999999999999e-05 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961517+0000", 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5296999999999999e-05 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961529+0000", 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1763000000000001e-05 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961621+0000", 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1636000000000002e-05 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961656+0000", 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5363e-05 2024-09-13T05:18:55.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966359+0000", 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047030839999999997 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966479+0000", 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119839 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.682462495000003, 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:18:55.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:18:55.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:18:55.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:18:55.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:18:55.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:18:55.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:18:55.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:18:55.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:18:55.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:18:55.831 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.832 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.832 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.832 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.832 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:18:55.832 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:55.833 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.681831236000001, 2024-09-13T05:18:55.833 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:18:55.833 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.833 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.833 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:55.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:55.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:18:55.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:55.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.838 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.681186115000003, 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.839 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:18:55.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.680678751999999, 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.844 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:18:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:18:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.680153316000002, 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.850 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:18:55.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.679515803999998, 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:18:55.852 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.854 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:18:55.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.856 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.678939096000001, 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.857 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.858 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:18:55.859 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:18:55.860 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.678387622999999, 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.861 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.862 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:18:55.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:18:55.864 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.67792704, 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:18:55.865 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.866 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:18:55.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.677417153999997, 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.870 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:18:55.873 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.676990091, 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.874 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:116 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'78, mlcod=14'65)", 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.676529404, 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050255040000000001, 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.878 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138458+0000", 2024-09-13T05:18:55.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138469+0000", 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1596e-05 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138490+0000", 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0446e-05 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.880 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138521+0000", 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0843000000000003e-05 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142091+0000", 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035703929999999998 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142146+0000", 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4873000000000001e-05 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.881 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143395+0000", 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012491169999999999 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143485+0000", 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9419000000000006e-05 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:117 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'79, mlcod=14'65)", 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:55.882 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.676037694999998, 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050956009999999999, 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.883 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138949+0000", 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138962+0000", 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2876e-05 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138987+0000", 2024-09-13T05:18:55.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4652999999999999e-05 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139021+0000", 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4122999999999998e-05 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142737+0000", 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003715472 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142773+0000", 2024-09-13T05:18:55.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6086e-05 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144004+0000", 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001231631 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144046+0000", 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1996000000000002e-05 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:132 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'90, mlcod=14'86)", 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.292850428999998, 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005226835, 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.887 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522137+0000", 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522142+0000", 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7040000000000002e-06 2024-09-13T05:18:55.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522153+0000", 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0573e-05 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522158+0000", 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9389999999999996e-06 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522191+0000", 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3571000000000001e-05 2024-09-13T05:18:55.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522204+0000", 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3169999999999999e-05 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527326+0000", 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00512153 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527365+0000", 2024-09-13T05:18:55.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8971999999999999e-05 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.290230373, 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.891 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:18:55.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:18:55.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:153 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'107, mlcod=14'105)", 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 44.673903672000002, 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046950569999999999, 2024-09-13T05:18:55.895 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141083+0000", 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141104+0000", 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0945999999999999e-05 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141127+0000", 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3054e-05 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141149+0000", 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1690000000000001e-05 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141266+0000", 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000117359 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141303+0000", 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7013e-05 2024-09-13T05:18:55.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145722+0000", 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044190260000000004 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145780+0000", 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7751999999999999e-05 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 44.672695560999998, 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.900 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:18:55.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:18:55.902 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:18:55.903 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:156 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'110, mlcod=14'105)", 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 44.672116688999999, 2024-09-13T05:18:55.904 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047886200000000004, 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142871+0000", 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142890+0000", 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9882999999999999e-05 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142944+0000", 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3465999999999999e-05 2024-09-13T05:18:55.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142959+0000", 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4644000000000001e-05 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144037+0000", 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001077988 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144094+0000", 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7079e-05 2024-09-13T05:18:55.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147601+0000", 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035074400000000001 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147660+0000", 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9243000000000002e-05 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:159 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'113, mlcod=14'105)", 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 44.670431589000003, 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044745929999999998, 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144556+0000", 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144583+0000", 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7416e-05 2024-09-13T05:18:55.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144632+0000", 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9453999999999998e-05 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144651+0000", 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8635999999999999e-05 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.146970+0000", 2024-09-13T05:18:55.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023192159999999998 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147003+0000", 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2595999999999998e-05 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.148955+0000", 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001951595 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:55.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.149031+0000", 2024-09-13T05:18:55.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6922999999999995e-05 2024-09-13T05:18:55.913 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.913 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.913 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.913 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:55.913 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:55.913 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:55.914 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:18:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:18:55.915 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:55.915 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:18:55.916 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:55.917 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:55.917 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:55.911+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:56.015 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:56.016+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:56.043 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:56.043 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:56.058 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:18:56.116 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:56.111+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:56.217 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:56.215+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:56.222 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:56.222 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:56.236 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:18:56.318 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:56.319+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:56.360 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:56.360 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:56.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:56.421 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:56.419+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:56.508 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:56.508 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:56.520 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:56.515+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:56.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:56.621 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:56.619+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:56.689 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:56.690 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:56.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:56.722 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:56.724+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:56.823 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:56.824+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:56.868 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:56.869 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:56.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:18:56.929 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:56.927+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:57.011 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:57.011 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:57.026 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:18:57.026 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:57.023+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:57.126 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:57.123+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:57.187 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:57.187 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:57.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:18:57.245 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:57.243+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:57.327 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:57.328+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:57.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:18:57.374 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:57.374 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:57.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:18:57.428 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:57.428+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:57.529 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:57.527+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:57.563 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:57.563 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:57.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:18:57.627 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 3] 2024-09-13T05:18:57.627 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:18:57.627 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T05:18:57.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T05:18:57.665 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:57.663+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:57.731 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:57.732+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:57.740 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:57.740 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:57.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:18:57.832 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:57.832+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:57.874 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:57.875 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:57.889 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:18:57.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:18:57.934 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:57.936+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:57.956 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:57.957 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:57.957 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 35, 2024-09-13T05:18:57.957 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5", 2024-09-13T05:18:57.957 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2024-09-13T05:17:57.363809+0000", 2024-09-13T05:18:57.957 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2024-09-13T05:18:55.319106+0000", 2024-09-13T05:18:57.957 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2024-09-13T05:18:42.246948+0000", 2024-09-13T05:18:57.957 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2024-09-13T05:17:59.738330+0000", 2024-09-13T05:18:57.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T05:18:57.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5804032, 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "nodeep-scrub", 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 6, 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T05:18:57.958 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "jewel", 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "reef", 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_crimson": false, 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": ".mgr", 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:08.122164+0000", 2024-09-13T05:18:57.959 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:18:57.960 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:18:57.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "15", 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:18:57.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:18:57.963 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:18:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": {} 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 6.059999942779541, 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 6.059999942779541, 2024-09-13T05:18:57.965 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.33000001311302185, 2024-09-13T05:18:57.966 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2, 2024-09-13T05:18:57.966 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2, 2024-09-13T05:18:57.966 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:18:57.966 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:18:57.966 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:18:57.966 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.966 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:15.001664+0000", 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:18:57.967 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:18:57.968 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:18:57.969 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:18:57.969 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:18:57.969 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:18:57.969 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:18:57.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.969 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "19", 2024-09-13T05:18:57.969 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:18:57.969 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:18:57.969 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 3, 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 19, 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:18:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:18:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:18:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 2.25, 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 2.25, 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 1, 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2.25, 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2.25, 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:18:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "e58174f4-85ec-4321-8bc3-384261c59c6b", 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 16, 2024-09-13T05:18:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:18:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:18:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2024-09-13T05:18:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:18:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/2153409514", 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/2153409514", 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/2153409514", 2024-09-13T05:18:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/2153409514", 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "84a3db6c-dacd-43d5-b964-37fdef0091a4", 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:18:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 16, 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:18:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:18:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/2608485151", 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/2608485151", 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/2608485151", 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/2608485151", 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:18:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "0d530476-cf38-4544-9ce7-e8ed35df220b", 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:18:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 16, 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2024-09-13T05:18:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:18:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/1006404377", 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/1006404377", 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/1006404377", 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/1006404377", 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "fd91f7e2-580d-47ea-9db5-792c8f1ebe01", 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:18:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.980804443359375, 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 20, 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 29, 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 29, 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 21, 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6816", 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6817", 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6818", 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6819", 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6822", 2024-09-13T05:18:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6823", 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6820", 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6821", 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6817/1834661236", 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6819/1834661236", 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6823/1834661236", 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6821/1834661236", 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "01af0f4c-5da2-46fe-a262-59fcc6a2efa9", 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:18:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:18:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:18:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:18:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:18:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:18:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 16, 2024-09-13T05:18:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:18:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:18:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:18:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6808", 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6809", 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:18:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6810", 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6811", 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6814", 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6815", 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6812", 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6813", 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6809/2630134185", 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6811/2630134185", 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6815/2630134185", 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6813/2630134185", 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "64cdd5f2-1568-4e96-ad90-6ef7414651f0", 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:18:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 29, 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6800", 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:18:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6801", 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6802", 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6803", 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6806", 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6807", 2024-09-13T05:18:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6804", 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6805", 2024-09-13T05:18:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6801/2778785304", 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6803/2778785304", 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6807/2778785304", 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6805/2778785304", 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.193869+0000", 2024-09-13T05:18:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.255764+0000", 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.286354+0000", 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:18:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.221310+0000", 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 21 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.026235+0000", 2024-09-13T05:18:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.010242+0000", 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:18:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [], 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_primaries": [], 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2024-09-13T05:18:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2024-09-13T05:18:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2024-09-13T05:18:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2024-09-13T05:18:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2024-09-13T05:18:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2024-09-13T05:18:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:58.018 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:58.019 INFO:tasks.thrashosds.thrasher:j is {'epoch': 35, 'fsid': '1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5', 'created': '2024-09-13T05:17:57.363809+0000', 'modified': '2024-09-13T05:18:55.319106+0000', 'last_up_change': '2024-09-13T05:18:42.246948+0000', 'last_in_change': '2024-09-13T05:17:59.738330+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T05:18:08.122164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T05:18:15.001664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e58174f4-85ec-4321-8bc3-384261c59c6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 2153409514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 2153409514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 2153409514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 2153409514}]}, 'public_addr': '172.21.15.16:6801/2153409514', 'cluster_addr': '172.21.15.16:6803/2153409514', 'heartbeat_back_addr': '172.21.15.16:6807/2153409514', 'heartbeat_front_addr': '172.21.15.16:6805/2153409514', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '84a3db6c-dacd-43d5-b964-37fdef0091a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 2608485151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 2608485151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 2608485151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 2608485151}]}, 'public_addr': '172.21.15.16:6809/2608485151', 'cluster_addr': '172.21.15.16:6811/2608485151', 'heartbeat_back_addr': '172.21.15.16:6815/2608485151', 'heartbeat_front_addr': '172.21.15.16:6813/2608485151', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0d530476-cf38-4544-9ce7-e8ed35df220b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 1006404377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 1006404377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 1006404377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 1006404377}]}, 'public_addr': '172.21.15.16:6817/1006404377', 'cluster_addr': '172.21.15.16:6819/1006404377', 'heartbeat_back_addr': '172.21.15.16:6823/1006404377', 'heartbeat_front_addr': '172.21.15.16:6821/1006404377', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fd91f7e2-580d-47ea-9db5-792c8f1ebe01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.980804443359375, 'last_clean_begin': 12, 'last_clean_end': 20, 'up_from': 29, 'up_thru': 29, 'down_at': 21, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6816', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6817', 'nonce': 1834661236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6818', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6819', 'nonce': 1834661236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6822', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6823', 'nonce': 1834661236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6820', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6821', 'nonce': 1834661236}]}, 'public_addr': '172.21.15.79:6817/1834661236', 'cluster_addr': '172.21.15.79:6819/1834661236', 'heartbeat_back_addr': '172.21.15.79:6823/1834661236', 'heartbeat_front_addr': '172.21.15.79:6821/1834661236', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '01af0f4c-5da2-46fe-a262-59fcc6a2efa9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6808', 'nonce': 2630134185}, {'type': 'v1', 'addr': '172.21.15.79:6809', 'nonce': 2630134185}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6810', 'nonce': 2630134185}, {'type': 'v1', 'addr': '172.21.15.79:6811', 'nonce': 2630134185}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6814', 'nonce': 2630134185}, {'type': 'v1', 'addr': '172.21.15.79:6815', 'nonce': 2630134185}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6812', 'nonce': 2630134185}, {'type': 'v1', 'addr': '172.21.15.79:6813', 'nonce': 2630134185}]}, 'public_addr': '172.21.15.79:6809/2630134185', 'cluster_addr': '172.21.15.79:6811/2630134185', 'heartbeat_back_addr': '172.21.15.79:6815/2630134185', 'heartbeat_front_addr': '172.21.15.79:6813/2630134185', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '64cdd5f2-1568-4e96-ad90-6ef7414651f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6800', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6801', 'nonce': 2778785304}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6802', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6803', 'nonce': 2778785304}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6806', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6807', 'nonce': 2778785304}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6804', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6805', 'nonce': 2778785304}]}, 'public_addr': '172.21.15.79:6801/2778785304', 'cluster_addr': '172.21.15.79:6803/2778785304', 'heartbeat_back_addr': '172.21.15.79:6807/2778785304', 'heartbeat_front_addr': '172.21.15.79:6805/2778785304', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.193869+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.255764+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.286354+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.221310+0000', 'dead_epoch': 21}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.026235+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.010242+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T05:18:58.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:18:58.020 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:18:58.020 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:18:58.034 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:18:58.034 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:58.036+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:58.054 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:18:58.054 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:18:58.070 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:18:58.070 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:18:58.086 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:18:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:18:58.103 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:18:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:18:58.122 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:18:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:18:58.134 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:58.136+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:58.138 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:18:58.138 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:18:58.194 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:58.194 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:58.194 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:58.194 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:58.194 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:58.209 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:18:58.249 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:58.252+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:18:58.330 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:18:58.336 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:58.336+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:18:58.351 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-13T05:18:58.060637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":178776,"kb_used_data":3268,"kb_used_omap":12,"kb_used_meta":175411,"kb_avail":566052264,"statfs":{"total":579820584960,"available":579637518336,"internally_reserved":0,"allocated":3346432,"data_stored":1940932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12399,"internal_metadata":179621777},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002196"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T05:18:49.292544+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:18:49.292544+0000","last_peered":"2024-09-13T05:18:49.292544+0000","last_clean":"2024-09-13T05:18:49.292544+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:18:49.292544+0000","last_undegraded":"2024-09-13T05:18:49.292544+0000","last_fullsized":"2024-09-13T05:18:49.292544+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T05:18:49.318650+0000","last_change":"2024-09-13T05:18:18.143477+0000","last_active":"2024-09-13T05:18:49.318650+0000","last_peered":"2024-09-13T05:18:49.318650+0000","last_clean":"2024-09-13T05:18:49.318650+0000","last_became_active":"2024-09-13T05:18:16.129462+0000","last_became_peered":"2024-09-13T05:18:16.129462+0000","last_unstale":"2024-09-13T05:18:49.318650+0000","last_undegraded":"2024-09-13T05:18:49.318650+0000","last_fullsized":"2024-09-13T05:18:49.318650+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:33:02.115421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076334099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":30,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T05:18:49.292542+0000","last_change":"2024-09-13T05:18:18.324354+0000","last_active":"2024-09-13T05:18:49.292542+0000","last_peered":"2024-09-13T05:18:49.292542+0000","last_clean":"2024-09-13T05:18:49.292542+0000","last_became_active":"2024-09-13T05:18:16.125619+0000","last_became_peered":"2024-09-13T05:18:16.125619+0000","last_unstale":"2024-09-13T05:18:49.292542+0000","last_undegraded":"2024-09-13T05:18:49.292542+0000","last_fullsized":"2024-09-13T05:18:49.292542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:40.629120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T05:18:49.318351+0000","last_change":"2024-09-13T05:18:43.275214+0000","last_active":"2024-09-13T05:18:49.318351+0000","last_peered":"2024-09-13T05:18:49.318351+0000","last_clean":"2024-09-13T05:18:49.318351+0000","last_became_active":"2024-09-13T05:18:43.274886+0000","last_became_peered":"2024-09-13T05:18:43.274886+0000","last_unstale":"2024-09-13T05:18:49.318351+0000","last_undegraded":"2024-09-13T05:18:49.318351+0000","last_fullsized":"2024-09-13T05:18:49.318351+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:53.129994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":33,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T05:18:49.316175+0000","last_change":"2024-09-13T05:18:18.147989+0000","last_active":"2024-09-13T05:18:49.316175+0000","last_peered":"2024-09-13T05:18:49.316175+0000","last_clean":"2024-09-13T05:18:49.316175+0000","last_became_active":"2024-09-13T05:18:16.129277+0000","last_became_peered":"2024-09-13T05:18:16.129277+0000","last_unstale":"2024-09-13T05:18:49.316175+0000","last_undegraded":"2024-09-13T05:18:49.316175+0000","last_fullsized":"2024-09-13T05:18:49.316175+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:59:39.922642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T05:18:52.315864+0000","last_change":"2024-09-13T05:18:43.272711+0000","last_active":"2024-09-13T05:18:52.315864+0000","last_peered":"2024-09-13T05:18:52.315864+0000","last_clean":"2024-09-13T05:18:52.315864+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:18:52.315864+0000","last_undegraded":"2024-09-13T05:18:52.315864+0000","last_fullsized":"2024-09-13T05:18:52.315864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:27.589949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T05:18:49.291340+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:18:49.291340+0000","last_peered":"2024-09-13T05:18:49.291340+0000","last_clean":"2024-09-13T05:18:49.291340+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:18:49.291340+0000","last_undegraded":"2024-09-13T05:18:49.291340+0000","last_fullsized":"2024-09-13T05:18:49.291340+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":31,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T05:18:49.292556+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:18:49.292556+0000","last_peered":"2024-09-13T05:18:49.292556+0000","last_clean":"2024-09-13T05:18:49.292556+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:18:49.292556+0000","last_undegraded":"2024-09-13T05:18:49.292556+0000","last_fullsized":"2024-09-13T05:18:49.292556+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T05:18:49.292556+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:18:49.292556+0000","last_peered":"2024-09-13T05:18:49.292556+0000","last_clean":"2024-09-13T05:18:49.292556+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:18:49.292556+0000","last_undegraded":"2024-09-13T05:18:49.292556+0000","last_fullsized":"2024-09-13T05:18:49.292556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607566,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":946176,"data_stored":714776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]}]},{"osd":4,"up_from":12,"seq":51539607566,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":128997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]}]},{"osd":3,"up_from":29,"seq":124554051588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28476,"kb_used_data":368,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94343364,"statfs":{"total":96636764160,"available":96607604736,"internally_reserved":0,"allocated":376832,"data_stored":133548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]}]},{"osd":2,"up_from":12,"seq":51539607565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":352256,"data_stored":124408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":1,"up_from":12,"seq":51539607565,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":946176,"data_stored":714776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":0,"up_from":12,"seq":51539607565,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31468,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340372,"statfs":{"total":96636764160,"available":96604540928,"internally_reserved":0,"allocated":356352,"data_stored":124427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:18:58.352 INFO:tasks.thrashosds.thrasher:Setting 2.5 to [2, 3, 0, 1] 2024-09-13T05:18:58.353 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.5', '2', '3', '0', '1'] 2024-09-13T05:18:58.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.5 2 3 0 1 2024-09-13T05:18:58.373 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:58.373 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:58.373 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:58.373 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:58.373 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.373 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-13T05:18:58.373 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.236625285000002, 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001669217, 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139958+0000", 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139957+0000", 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139968+0000", 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.007e-05 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.139993+0000", 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5945999999999999e-05 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140008+0000", 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4926000000000001e-05 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140092+0000", 2024-09-13T05:18:58.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3282999999999995e-05 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.140137+0000", 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5195999999999999e-05 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141553+0000", 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014160659999999999 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:17.141627+0000", 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4330000000000002e-05 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4190.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.226921089999998, 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001729281, 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149662+0000", 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:58.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149679+0000", 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7090000000000001e-05 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149706+0000", 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7192999999999999e-05 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149721+0000", 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.45e-05 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149824+0000", 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000102959 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.149862+0000", 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8667000000000001e-05 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151293+0000", 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001431332 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:18.151392+0000", 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8023000000000007e-05 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.382 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:58.387 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:18:58.473 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:58.476+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:58.538 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:58.540+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:58.546 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:58.546 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:58.546 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:58.546 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:58.560 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:18:58.639 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:58.635+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:58.726 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:58.726 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:58.726 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:58.726 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:58.726 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:58.740 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:18:58.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:58.739+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:58.841 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:58.844+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:58.864 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:58.864 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'53)", 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.903533578999998, 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050448339999999998, 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.865 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.866 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961433+0000", 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961492+0000", 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9488999999999999e-05 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961517+0000", 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5296999999999999e-05 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961529+0000", 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1763000000000001e-05 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961621+0000", 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1636000000000002e-05 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961656+0000", 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5363e-05 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966359+0000", 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047030839999999997 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966479+0000", 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119839 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.869 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.732441823000002, 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:58.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:18:58.874 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.731810564, 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:18:58.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:18:58.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.731165443000002, 2024-09-13T05:18:58.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:18:58.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.881 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:18:58.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.884 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.884 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:18:58.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.730658079999998, 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.887 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.888 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:18:58.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.730132644000001, 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:18:58.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.894 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.895 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.729495131999997, 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.896 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:18:58.900 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.728918424, 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.901 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.902 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:18:58.903 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.904 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:18:58.905 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:18:58.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.905 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.905 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:18:58.905 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:18:58.905 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.905 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.905 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.728366950999998, 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:58.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:18:58.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:58.910 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.727906367999999, 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:18:58.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.914 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.727396482000003, 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.915 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:18:58.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:18:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.726969419, 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:18:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:18:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:116 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'78, mlcod=14'65)", 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.726508731999999, 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050255040000000001, 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138458+0000", 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138469+0000", 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1596e-05 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138490+0000", 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0446e-05 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138521+0000", 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0843000000000003e-05 2024-09-13T05:18:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142091+0000", 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035703929999999998 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142146+0000", 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4873000000000001e-05 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143395+0000", 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012491169999999999 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143485+0000", 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9419000000000006e-05 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:117 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'79, mlcod=14'65)", 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.726017022999997, 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050956009999999999, 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138949+0000", 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138962+0000", 2024-09-13T05:18:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2876e-05 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138987+0000", 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4652999999999999e-05 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139021+0000", 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4122999999999998e-05 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142737+0000", 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003715472 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142773+0000", 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6086e-05 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144004+0000", 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001231631 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144046+0000", 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1996000000000002e-05 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:132 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'90, mlcod=14'86)", 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.342829756999997, 2024-09-13T05:18:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005226835, 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522137+0000", 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522142+0000", 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7040000000000002e-06 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522153+0000", 2024-09-13T05:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0573e-05 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522158+0000", 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9389999999999996e-06 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522191+0000", 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3571000000000001e-05 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522204+0000", 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3169999999999999e-05 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527326+0000", 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00512153 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527365+0000", 2024-09-13T05:18:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8971999999999999e-05 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 48.340209700999999, 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:18:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:153 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'107, mlcod=14'105)", 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.723883000000001, 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046950569999999999, 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:18:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141083+0000", 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141104+0000", 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0945999999999999e-05 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141127+0000", 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3054e-05 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141149+0000", 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1690000000000001e-05 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141266+0000", 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000117359 2024-09-13T05:18:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141303+0000", 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7013e-05 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145722+0000", 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044190260000000004 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145780+0000", 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7751999999999999e-05 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.722674888999997, 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:18:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:18:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:156 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'110, mlcod=14'105)", 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.722096016999998, 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047886200000000004, 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142871+0000", 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142890+0000", 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9882999999999999e-05 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142944+0000", 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3465999999999999e-05 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142959+0000", 2024-09-13T05:18:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4644000000000001e-05 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144037+0000", 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001077988 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144094+0000", 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7079e-05 2024-09-13T05:18:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147601+0000", 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035074400000000001 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147660+0000", 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9243000000000002e-05 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:159 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'113, mlcod=14'105)", 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.720410917000002, 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044745929999999998, 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144556+0000", 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144583+0000", 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7416e-05 2024-09-13T05:18:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144632+0000", 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9453999999999998e-05 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144651+0000", 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8635999999999999e-05 2024-09-13T05:18:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.146970+0000", 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023192159999999998 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147003+0000", 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2595999999999998e-05 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.148955+0000", 2024-09-13T05:18:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001951595 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.149031+0000", 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6922999999999995e-05 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:18:58.963 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:58.965 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:18:58.965 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:58.964+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:59.065 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:18:59.063+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:18:59.136 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:59.136 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:59.137 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:18:59.137 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:59.150 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:18:59.198 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:59.200+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:59.267 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:18:59.268+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:18:59.268 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:59.268 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:18:59.268 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:18:59.268 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:18:59.268 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:59.282 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:18:59.344 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.5 pg_upmap_items mapping to [2->3,0->1] 2024-09-13T05:18:59.409 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:59.407+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:18:59.450 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:18:59.450 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:18:59.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:18:59.451 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [] 2024-09-13T05:18:59.451 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:18:59.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:18:59.469 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:59.472+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:59.570 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:59.572+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:59.600 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:59.600 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:59.600 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:18:59.600 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:59.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:18:59.671 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:18:59.672+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:18:59.731 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:59.731 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:18:59.731 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:18:59.731 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:18:59.731 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:59.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:18:59.793 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:18:59.791+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:18:59.873 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:18:59.872+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:18:59.875 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:18:59.875 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:18:59.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:18:59.875 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:18:59.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.875 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:59.875 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.737102151000002, 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:59.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:18:59.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.879 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.736462181999997, 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:59.880 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:59.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:18:59.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.881 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:59.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:59.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:59.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:18:59.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:18:59.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:59.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:18:59.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:18:59.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:59.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.741589120999997, 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:59.886 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:18:59.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:18:59.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:18:59.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.893 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:18:59.894 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:59.894 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.735477328999998, 2024-09-13T05:18:59.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:18:59.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:59.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:59.894 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:59.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:18:59.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:59.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.731807590999999, 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:18:59.898 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:59.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:18:59.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:18:59.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:59.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:18:59.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:18:59.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:18:59.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:18:59.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:18:59.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:18:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:18:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:18:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:18:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:59.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.725669871999997, 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:18:59.906 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:18:59.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:18:59.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:59.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:18:59.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:18:59.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:18:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:18:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:18:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:18:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:18:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:18:59.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:18:59.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:18:59.911 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:18:59.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:00.001 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:18:59.999+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:00.075 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:00.071+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:00.082 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:00.083 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:00.083 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:00.083 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:00.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:00.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:00.176 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:00.176+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:00.272 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:00.273 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:00.273 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:00.273 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:00.273 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:00.276 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:00.276+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:00.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:00.336 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:00.336 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:00.353 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:00.358 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:19:00.401 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:00.399+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:00.415 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:00.416 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:00.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:00.478 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:00.480+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:00.579 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:00.580+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:19:00.590 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:00.591 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:00.591 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:00.591 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:00.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:00.616 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:00.616 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:00.649 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:00.650 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:00.670 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:00.670 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:00.680 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:00.680+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:00.721 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:00.722 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:00.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:00.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:00.780+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:00.882 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:00.879+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:00.905 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:00.905 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:00.918 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:00.983 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:00.984+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:01.049 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:01.049 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:01.063 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:01.106 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:01.108+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:01.184 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:01.184+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:01.222 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:01.222 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:01.236 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:01.322 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:01.324+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:19:01.386 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:01.383+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:01.402 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:01.402 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:01.417 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:01.505 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:01.508+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:01.588 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:01.588 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:01.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:01.588+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:01.601 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:01.690 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:01.692+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:01.766 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:01.766 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:01.781 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:01.826 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:01.828+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:19:01.891 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:01.892+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:01.948 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:01.949 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:01.962 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:01.992 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:01.987+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:02.093 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:02.092+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:19:02.130 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:02.130 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:02.144 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:02.230 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:02.231+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:19:02.294 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:02.295+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:02.309 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:02.309 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:02.324 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:02.395 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:02.391+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:02.488 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:02.488 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:02.496 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:02.495+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:02.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:02.598 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:02.595+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:02.635 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:02.635 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:02.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:02.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:02.737 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:02.735+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:02.800 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:02.795+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:02.818 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:02.818 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:02.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:02.833 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:02.833 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:02.848 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:02.865 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:02.885 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:02.885 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:02.900 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:02.895+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:02.905 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:02.925 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:02.925 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:02.964 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:19:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.826362797000002, 2024-09-13T05:19:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:19:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:19:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.825722827999996, 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:19:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.830849766999997, 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:19:02.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:19:02.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:02.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:19:02.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:19:02.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:19:02.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.824737974999998, 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:19:02.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:02.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.821068236999999, 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:02.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:19:02.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:19:02.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:02.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.814930517999997, 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:19:02.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:02.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.996 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:02.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:02.997 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:02.997 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:02.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:03.001 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:03.003+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:03.102 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:03.103+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:03.170 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:03.170 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:03.170 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:03.170 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:03.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:03.203 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:03.203+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:03.303 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:03.304 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:03.304 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:03.304 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:03.304 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:03.304 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:03.299+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:03.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:03.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:19:03.405 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:03.403+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:03.496 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:03.496 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:03.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:03.496 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:03.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.496 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.496 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:19:03.496 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.635310304999997, 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010082459, 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:19:03.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858216+0000", 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858229+0000", 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.305e-05 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858266+0000", 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7523e-05 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858283+0000", 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6597000000000001e-05 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865356+0000", 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070732859999999998 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865490+0000", 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133442 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865698+0000", 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208202 2024-09-13T05:19:03.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867616+0000", 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019181409999999999 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868169+0000", 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055310299999999999 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868246+0000", 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6866000000000006e-05 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868300+0000", 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3912999999999997e-05 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.635222099000003, 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010336577, 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:19:03.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858304+0000", 2024-09-13T05:19:03.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858318+0000", 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3963e-05 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858337+0000", 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8482999999999999e-05 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858351+0000", 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4195999999999999e-05 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866611+0000", 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082601199999999993 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866763+0000", 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015145600000000001 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866987+0000", 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224441 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867784+0000", 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079658099999999998 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868531+0000", 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074704700000000003 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868596+0000", 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4966000000000001e-05 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868642+0000", 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6607000000000003e-05 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.541323536999997, 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:03.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:03.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:19:03.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:19:03.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:19:03.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:19:03.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:19:03.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:19:03.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:19:03.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:19:03.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:19:03.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:19:03.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.516 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.541116379000002, 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.517 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:19:03.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.363877649999999, 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010071422, 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.524 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.524 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.524 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.524 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 107 2024-09-13T05:19:03.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.524 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:19:03.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:19:03.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129649+0000", 2024-09-13T05:19:03.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129656+0000", 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129668+0000", 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2463000000000001e-05 2024-09-13T05:19:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129675+0000", 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1999999999999997e-06 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131247+0000", 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015719359999999999 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131321+0000", 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4232000000000003e-05 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131413+0000", 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1525000000000002e-05 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.133468+0000", 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002055177 2024-09-13T05:19:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139645+0000", 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061772490000000001 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139691+0000", 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5967000000000002e-05 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139722+0000", 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0273e-05 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.363133425000001, 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.531 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:03.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:19:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:19:03.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.537 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.363004553000003, 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.543 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.362719261000002, 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010554714999999999, 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 115 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.544 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130808+0000", 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130882+0000", 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4449999999999994e-05 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130891+0000", 2024-09-13T05:19:03.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7190000000000007e-06 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130897+0000", 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135420+0000", 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045232060000000001 2024-09-13T05:19:03.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135472+0000", 2024-09-13T05:19:03.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1530999999999998e-05 2024-09-13T05:19:03.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135557+0000", 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5192999999999996e-05 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136761+0000", 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001204072 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141283+0000", 2024-09-13T05:19:03.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045216099999999997 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141331+0000", 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8094999999999999e-05 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141363+0000", 2024-09-13T05:19:03.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2319e-05 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.362624994000001, 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010559903000000001, 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 116 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130902+0000", 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130905+0000", 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130909+0000", 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7239999999999998e-06 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130914+0000", 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1800000000000004e-06 2024-09-13T05:19:03.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135956+0000", 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050418850000000003 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136008+0000", 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1694999999999999e-05 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136090+0000", 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2527000000000005e-05 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137204+0000", 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011136640000000001 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141406+0000", 2024-09-13T05:19:03.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042020039999999996 2024-09-13T05:19:03.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141438+0000", 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2277999999999998e-05 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141463+0000", 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4396000000000002e-05 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.558 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.362607138999998, 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:03.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:19:03.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:19:03.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:19:03.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:19:03.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.978226816000003, 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:03.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:19:03.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.97818968, 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:19:03.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:19:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.577 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.978083761000001, 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:03.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:03.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:03.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:03.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:03.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:03.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:03.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.977811736, 2024-09-13T05:19:03.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:03.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.586 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:03.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.586 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:03.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:03.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:03.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:03.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.977768150000003, 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:03.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:03.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:03.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:03.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:03.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:03.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:03.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:03.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:03.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:03.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:03.600 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.977735922999997, 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:03.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:03.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:03.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.977524774999999, 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:03.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:03.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.612 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.977488481999998, 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.613 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:03.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.977450032, 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:03.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:03.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:03.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:03.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:03.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:03.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:03.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:03.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:03.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.626 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.626 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:03.626 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:03.626 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.977275130000002, 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:03.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:03.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:03.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:03.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:03.633 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:03.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:03.635 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:03.631+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:03.736 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:03.731+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:03.811 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:03.811 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:03.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:03.838 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:03.839+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:19:03.939 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:03.935+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:03.993 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:03.993 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:03.993 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:03.993 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:03.993 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:04.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:04.049 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:04.047+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:04.141 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:04.139+0000 7f162d1b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25193) UID: 0 2024-09-13T05:19:04.162 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:04.162 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:04.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:04.163 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:04.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.163 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:19:04.163 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:04.163 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 48.030844084000002, 2024-09-13T05:19:04.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:19:04.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:04.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:04.163 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:19:04.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:04.167 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:04.168 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:04.168 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:04.168 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:04.168 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:04.176 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:04.242 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:04.239+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:04.307 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:04.307 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:04.307 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:04.321 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:04.343 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:04.339+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:04.366 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:19:04.367 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 3] 2024-09-13T05:19:04.367 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:19:04.367 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 4, 5, 3] 2024-09-13T05:19:04.368 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T05:19:04.368 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:19:04.444 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:04.439+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:04.445 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: exception: [Errno 104] Connection reset by peer 2024-09-13T05:19:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T05:19:04.449 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:04.453 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T05:19:04.453 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-09-13T05:19:04.453 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:19:04.453 DEBUG:teuthology.orchestra.run.smithi079:> type ceph-objectstore-tool 2024-09-13T05:19:04.501 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T05:19:04.502 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-09-13T05:19:04.544 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:04.543+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-13T05:19:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T05:19:04.559 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:04.645 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:04.643+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:04.717 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:04.717 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:04.717 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:04.717 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:04.731 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:04.778 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:04.779+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:04.847 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:04.843+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:04.851 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:04.851 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:04.851 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:04.851 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:04.852 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:04.866 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:04.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:04.958 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:04.959+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:05.028 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:05.028 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4201.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.93408822499999999, 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00040690000000000002, 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4201", 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.79:0/17281544", 2024-09-13T05:19:05.029 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 2 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:05.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097544+0000", 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097558+0000", 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3991999999999999e-05 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097581+0000", 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3238e-05 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:05.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097598+0000", 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7470999999999999e-05 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097717+0000", 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000118991 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097953+0000", 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023558699999999999 2024-09-13T05:19:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:05.033 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:05.033 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:05.033 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:05.033 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:05.033 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:05.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:05.090 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:05.091+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:05.100 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:05.101 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:05.121 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:05.121 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:05.136 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:05.136 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:05.149 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:05.147+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:05.156 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:05.156 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:05.169 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:05.169 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:05.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:05.273 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:05.271+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:05.348 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:05.349 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:05.351 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:05.347+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:05.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:05.452 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:05.451+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:05.532 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:05.532 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:05.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:05.553 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:05.555+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:05.654 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:05.651+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:05.666 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:05.666 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:05.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:05.711 INFO:teuthology.orchestra.run.smithi079.stdout:2.6 2024-09-13T05:19:05.711 INFO:teuthology.orchestra.run.smithi079.stdout:2.5 2024-09-13T05:19:05.711 INFO:teuthology.orchestra.run.smithi079.stdout:2.2 2024-09-13T05:19:05.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:05.763+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:05.847 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:05.847 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:05.855 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:05.855+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:05.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:05.956 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:05.955+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:06.037 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:06.038 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:06.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:06.057 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:06.051+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:06.157 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:06.155+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:06.181 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:06.181 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:06.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:06.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 2.6 --file /var/log/ceph/exp.2.6.4 2024-09-13T05:19:06.260 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:06.255+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:06.322 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:06.323 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:06.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:06.381 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:06.379+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:06.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:19:06.459 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:06.459+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:06.502 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:06.502 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:06.517 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:06.566 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:06.567+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:06.653 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:06.653 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:06.653 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:06.653 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:06.662 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:06.659+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:06.676 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:06.763 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:06.759+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:06.834 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:06.834 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:06.834 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:06.834 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:06.834 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:06.847 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:06.894 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:06.895+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'53)", 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.995553283, 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050448339999999998, 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.956 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961433+0000", 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961492+0000", 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9488999999999999e-05 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961517+0000", 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5296999999999999e-05 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961529+0000", 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1763000000000001e-05 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961621+0000", 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1636000000000002e-05 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961656+0000", 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5363e-05 2024-09-13T05:19:06.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966359+0000", 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047030839999999997 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966479+0000", 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119839 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.824461526999997, 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:06.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.823830268000002, 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.823185146999997, 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.969 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:06.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:06.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:06.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.822677784, 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.977 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.822152348000003, 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:06.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.821514835999999, 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.982 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:06.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.986 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.820938128000002, 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:06.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:06.990 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.820386655, 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:06.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.819926072000001, 2024-09-13T05:19:06.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:06.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:06.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.819416185999998, 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:07.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:07.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.818989123000001, 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:07.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:116 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'78, mlcod=14'65)", 2024-09-13T05:19:07.008 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.818528436000001, 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050255040000000001, 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138458+0000", 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138469+0000", 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1596e-05 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138490+0000", 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0446e-05 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138521+0000", 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0843000000000003e-05 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142091+0000", 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035703929999999998 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142146+0000", 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4873000000000001e-05 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143395+0000", 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012491169999999999 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143485+0000", 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9419000000000006e-05 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.012 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:117 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'79, mlcod=14'65)", 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.818036726999999, 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050956009999999999, 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:19:07.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138949+0000", 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138962+0000", 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2876e-05 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138987+0000", 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4652999999999999e-05 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139021+0000", 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4122999999999998e-05 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142737+0000", 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003715472 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142773+0000", 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6086e-05 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144004+0000", 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001231631 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144046+0000", 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1996000000000002e-05 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:132 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'90, mlcod=14'86)", 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.434849460999999, 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005226835, 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.017 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522137+0000", 2024-09-13T05:19:07.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522142+0000", 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7040000000000002e-06 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522153+0000", 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0573e-05 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522158+0000", 2024-09-13T05:19:07.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9389999999999996e-06 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522191+0000", 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3571000000000001e-05 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522204+0000", 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3169999999999999e-05 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527326+0000", 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00512153 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527365+0000", 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8971999999999999e-05 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:07.021 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.432229405000001, 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:07.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:153 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'107, mlcod=14'105)", 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 55.815902704000003, 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046950569999999999, 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141083+0000", 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141104+0000", 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0945999999999999e-05 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141127+0000", 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3054e-05 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141149+0000", 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1690000000000001e-05 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141266+0000", 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000117359 2024-09-13T05:19:07.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141303+0000", 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7013e-05 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145722+0000", 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044190260000000004 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145780+0000", 2024-09-13T05:19:07.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7751999999999999e-05 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 55.814694592999999, 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:07.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:07.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.034 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:156 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'110, mlcod=14'105)", 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 55.814115721, 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047886200000000004, 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142871+0000", 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142890+0000", 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9882999999999999e-05 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142944+0000", 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3465999999999999e-05 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142959+0000", 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4644000000000001e-05 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144037+0000", 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001077988 2024-09-13T05:19:07.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144094+0000", 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7079e-05 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147601+0000", 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035074400000000001 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147660+0000", 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9243000000000002e-05 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:159 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'113, mlcod=14'105)", 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 55.812430620999997, 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044745929999999998, 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:07.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144556+0000", 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144583+0000", 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7416e-05 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144632+0000", 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9453999999999998e-05 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144651+0000", 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8635999999999999e-05 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.146970+0000", 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023192159999999998 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147003+0000", 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2595999999999998e-05 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.148955+0000", 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001951595 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:07.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:07.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:07.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.149031+0000", 2024-09-13T05:19:07.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6922999999999995e-05 2024-09-13T05:19:07.043 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.043 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.043 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.043 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:07.043 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:07.043 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:07.045 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:07.045 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:07.043+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:07.146 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:07.143+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:07.208 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:07.208 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:07.221 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:07.247 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:07.243+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:07.337 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:07.337 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:07.347 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:07.347+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:07.350 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:19:07.449 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:07.447+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:07.460 INFO:teuthology.orchestra.run.smithi079.stderr:Exporting 2.6 info 2.6( empty local-lis/les=16/17 n=0 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) 2024-09-13T05:19:07.460 INFO:teuthology.orchestra.run.smithi079.stderr:Export successful 2024-09-13T05:19:07.521 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:07.521 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:07.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:07.582 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:07.579+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:07.650 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:07.647+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:07.670 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:07.670 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:07.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:07.751 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:07.751+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:07.852 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:07.847+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:07.857 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:07.857 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:07.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:07.957 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:07.955+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:07.995 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 2.6 2024-09-13T05:19:08.036 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:08.036 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:08.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:08.096 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:08.091+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:08.154 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:08.151+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:08.224 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:08.224 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:08.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:08.256 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:08.255+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:08.357 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:08.355+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:08.408 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:08.409 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:08.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:08.473 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:08.470+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:08.549 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:08.549 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:08.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:08.554+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:08.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:08.660 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:08.654+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:08.725 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:08.725 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:08.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:08.761 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:08.759+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:08.861 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:08.858+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:08.922 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:08.922 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:08.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:08.977 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:08.974+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:09.063 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:09.063+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:09.105 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:09.105 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:09.119 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:09.165 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:09.158+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:09.192 INFO:teuthology.orchestra.run.smithi079.stdout: marking collection for removal 2024-09-13T05:19:09.192 INFO:teuthology.orchestra.run.smithi079.stdout:setting '_remove' omap key 2024-09-13T05:19:09.193 INFO:teuthology.orchestra.run.smithi079.stdout:finish_remove_pgs 2.6_head removing 2.6 2024-09-13T05:19:09.196 INFO:teuthology.orchestra.run.smithi079.stdout:Remove successful 2024-09-13T05:19:09.252 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:09.253 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:09.253 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:09.253 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:09.265 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:09.263+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:09.266 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:09.366 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:09.362+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:09.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:19:09.432 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:09.433 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:09.433 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:09.433 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:09.433 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:09.447 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:09.494 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:09.495+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:09.568 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:09.562+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'53)", 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.609263245000001, 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050448339999999998, 2024-09-13T05:19:09.571 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961433+0000", 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961492+0000", 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9488999999999999e-05 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961517+0000", 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5296999999999999e-05 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961529+0000", 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1763000000000001e-05 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961621+0000", 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1636000000000002e-05 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961656+0000", 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5363e-05 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966359+0000", 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047030839999999997 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966479+0000", 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119839 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.575 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.438171488999998, 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.437540230000003, 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.580 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:09.584 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.436895108999998, 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:09.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:09.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.436387746000001, 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:09.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:09.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.435862309999997, 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:09.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.435224798, 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:09.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.434648090000003, 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.602 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.606 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.434096617000002, 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.433636034000003, 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.433126147999999, 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:09.619 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.432699085000003, 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:09.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.623 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:116 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'78, mlcod=14'65)", 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.432238398000003, 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050255040000000001, 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:19:09.624 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138458+0000", 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138469+0000", 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1596e-05 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.625 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138490+0000", 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0446e-05 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138521+0000", 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0843000000000003e-05 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142091+0000", 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035703929999999998 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142146+0000", 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4873000000000001e-05 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143395+0000", 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012491169999999999 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143485+0000", 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9419000000000006e-05 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.627 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:117 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'79, mlcod=14'65)", 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.431746689000001, 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050956009999999999, 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138949+0000", 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138962+0000", 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2876e-05 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138987+0000", 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4652999999999999e-05 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139021+0000", 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4122999999999998e-05 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142737+0000", 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003715472 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142773+0000", 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6086e-05 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144004+0000", 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001231631 2024-09-13T05:19:09.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144046+0000", 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1996000000000002e-05 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:132 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'90, mlcod=14'86)", 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.048559423, 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005226835, 2024-09-13T05:19:09.632 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522137+0000", 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522142+0000", 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7040000000000002e-06 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522153+0000", 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0573e-05 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522158+0000", 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9389999999999996e-06 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522191+0000", 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3571000000000001e-05 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522204+0000", 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3169999999999999e-05 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527326+0000", 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00512153 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527365+0000", 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8971999999999999e-05 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.045939367000003, 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:09.640 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:153 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'107, mlcod=14'105)", 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 58.429612665999997, 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046950569999999999, 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141083+0000", 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141104+0000", 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0945999999999999e-05 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141127+0000", 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3054e-05 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141149+0000", 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1690000000000001e-05 2024-09-13T05:19:09.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141266+0000", 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000117359 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141303+0000", 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7013e-05 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145722+0000", 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044190260000000004 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145780+0000", 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7751999999999999e-05 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.645 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 58.428404555, 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:09.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.650 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:156 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'110, mlcod=14'105)", 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 58.427825683000002, 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047886200000000004, 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142871+0000", 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142890+0000", 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9882999999999999e-05 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142944+0000", 2024-09-13T05:19:09.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3465999999999999e-05 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142959+0000", 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4644000000000001e-05 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144037+0000", 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001077988 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144094+0000", 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7079e-05 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147601+0000", 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035074400000000001 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147660+0000", 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9243000000000002e-05 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.654 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:159 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'113, mlcod=14'105)", 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 58.426140582999999, 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044745929999999998, 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144556+0000", 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144583+0000", 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7416e-05 2024-09-13T05:19:09.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144632+0000", 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9453999999999998e-05 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144651+0000", 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8635999999999999e-05 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.146970+0000", 2024-09-13T05:19:09.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023192159999999998 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147003+0000", 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2595999999999998e-05 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.148955+0000", 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001951595 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.149031+0000", 2024-09-13T05:19:09.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6922999999999995e-05 2024-09-13T05:19:09.659 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.659 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.659 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.659 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:09.659 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:09.659 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:09.660 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:09.668 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:09.662+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:09.760 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.6.4 2024-09-13T05:19:09.769 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:09.766+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:09.835 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:09.835 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:09.849 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:09.870 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:09.866+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:09.964 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:09.965 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:09.971 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:09.971+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:09.978 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:10.072 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:10.071+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:10.119 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:10.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:10.145 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:10.145 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:10.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:10.214 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:10.210+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:10.274 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:10.270+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:10.302 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:10.302 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:10.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:10.376 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:10.375+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:10.403 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:19:10.477 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:10.479+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:10.491 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:10.491 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:10.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:10.578 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:10.579+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:10.631 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:10.631 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:10.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:10.689 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:10.686+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:10.771 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:10.772 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:10.780 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:10.779+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:10.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:10.881 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:10.878+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:10.916 INFO:teuthology.orchestra.run.smithi079.stdout:get_pg_num_history pg_num_history pg_num_history(e38 pg_nums {1={13=1},2={16=8}} deleted_pools ) 2024-09-13T05:19:10.917 INFO:teuthology.orchestra.run.smithi079.stdout:Importing pgid 2.6 2024-09-13T05:19:10.917 INFO:teuthology.orchestra.run.smithi079.stdout:write_pg epoch 20 info 2.6( empty local-lis/les=16/17 n=0 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) 2024-09-13T05:19:10.920 INFO:teuthology.orchestra.run.smithi079.stdout:Import successful 2024-09-13T05:19:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:10.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:11.021 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:11.018+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:11.083 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:11.078+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:11.148 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:11.148 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:11.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:11.205 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:11.202+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:11.285 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:11.282+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:11.330 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:11.330 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:11.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:11.389 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:11.386+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:11.455 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /var/log/ceph/exp.2.6.4 2024-09-13T05:19:11.487 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:11.482+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:11.517 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:11.517 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:11.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:11.618 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:11.614+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:11.689 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:11.686+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:11.701 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:11.701 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:11.716 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:11.791 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:11.791+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:11.845 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:11.845 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:11.860 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:11.892 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:11.886+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:11.993 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:11.986+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:12.026 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:12.026 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:12.041 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:12.093 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:12.090+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:12.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:12.163 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:12.163 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:12.177 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:12.222 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:12.223+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:12.284 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:12.293 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:12.293 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:12.295 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:12.295+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:12.301 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:12.301 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:12.307 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:12.317 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:12.335 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:12.335 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:12.335 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:12.355 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:12.355 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:12.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:12.397 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:12.394+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:12.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:19:12.471 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:12.471 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:12.471 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:12.471 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:12.472 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:12.485 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:12.530 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:12.531+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:12.599 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:12.599+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:12.606 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4201.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.5116693770000005, 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00040690000000000002, 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:12.607 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4201", 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.79:0/17281544", 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 2 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097544+0000", 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097558+0000", 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3991999999999999e-05 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:12.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097581+0000", 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3238e-05 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097598+0000", 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7470999999999999e-05 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097717+0000", 2024-09-13T05:19:12.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000118991 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097953+0000", 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023558699999999999 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:12.611 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:12.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:12.700 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:12.694+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:12.742 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:12.742 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:12.742 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:12.742 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:12.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:12.841 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:12.838+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:12.902 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:12.898+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:12.921 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:12.921 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:12.921 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:12.921 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:12.921 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:12.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:13.004 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:13.002+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:13.104 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.966020249000003, 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.105 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:19:13.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:19:13.109 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.965380279999998, 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:19:13.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.970507218999998, 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:13.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:19:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.120 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.964395426999999, 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:13.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:19:13.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:19:13.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.960725689, 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:13.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:19:13.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.954587969999999, 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:19:13.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:19:13.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:19:13.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4201.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:04.092330+0000", 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.0114999119999997, 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00031489000000000002, 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4201", 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/17281544", 2024-09-13T05:19:13.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092330+0000", 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092330+0000", 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092328+0000", 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092342+0000", 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4596000000000001e-05 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092364+0000", 2024-09-13T05:19:13.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2229e-05 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092383+0000", 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8692e-05 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092473+0000", 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9740999999999993e-05 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092645+0000", 2024-09-13T05:19:13.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017189199999999999 2024-09-13T05:19:13.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.140 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.140 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.140 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:13.140 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:13.098+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:13.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:13.230 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:13.226+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:13.306 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:13.306+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:13.317 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:13.317 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:13.317 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:13.317 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:13.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:19:13.410 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:13.406+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:13.497 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:13.497 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:13.497 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:13.497 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:13.497 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:13.509 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:13.506+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:13.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:13.610 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:13.606+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.777562840000002, 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010082459, 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858216+0000", 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858229+0000", 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.305e-05 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858266+0000", 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7523e-05 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858283+0000", 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6597000000000001e-05 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865356+0000", 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070732859999999998 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865490+0000", 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133442 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865698+0000", 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208202 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867616+0000", 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019181409999999999 2024-09-13T05:19:13.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868169+0000", 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055310299999999999 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868246+0000", 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6866000000000006e-05 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868300+0000", 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3912999999999997e-05 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.777474634000001, 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010336577, 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858304+0000", 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858318+0000", 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3963e-05 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858337+0000", 2024-09-13T05:19:13.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8482999999999999e-05 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858351+0000", 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4195999999999999e-05 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866611+0000", 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082601199999999993 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866763+0000", 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015145600000000001 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866987+0000", 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224441 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867784+0000", 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079658099999999998 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868531+0000", 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074704700000000003 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868596+0000", 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4966000000000001e-05 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868642+0000", 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6607000000000003e-05 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.683576072000001, 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:19:13.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.656 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.683368913999999, 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.657 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.506130185000004, 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010071422, 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 107 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129649+0000", 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129656+0000", 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129668+0000", 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2463000000000001e-05 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129675+0000", 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1999999999999997e-06 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131247+0000", 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015719359999999999 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131321+0000", 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4232000000000003e-05 2024-09-13T05:19:13.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131413+0000", 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1525000000000002e-05 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.133468+0000", 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002055177 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139645+0000", 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061772490000000001 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139691+0000", 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5967000000000002e-05 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139722+0000", 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0273e-05 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.505385959999998, 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.669 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:19:13.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.505257088, 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:19:13.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:19:13.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:19:13.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:19:13.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.504971796, 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010554714999999999, 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 115 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130808+0000", 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130882+0000", 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4449999999999994e-05 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130891+0000", 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7190000000000007e-06 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130897+0000", 2024-09-13T05:19:13.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135420+0000", 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045232060000000001 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135472+0000", 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1530999999999998e-05 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135557+0000", 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5192999999999996e-05 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136761+0000", 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001204072 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141283+0000", 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045216099999999997 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141331+0000", 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8094999999999999e-05 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141363+0000", 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2319e-05 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.687 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.504877528999998, 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010559903000000001, 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 116 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:19:13.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130902+0000", 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130905+0000", 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130909+0000", 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7239999999999998e-06 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130914+0000", 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1800000000000004e-06 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135956+0000", 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050418850000000003 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136008+0000", 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1694999999999999e-05 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136090+0000", 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2527000000000005e-05 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137204+0000", 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011136640000000001 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141406+0000", 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042020039999999996 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141438+0000", 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2277999999999998e-05 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141463+0000", 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4396000000000002e-05 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.504859674000002, 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:19:13.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:19:13.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:19:13.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.120479351, 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:19:13.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:19:13.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:19:13.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:19:13.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:19:13.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.120442214999997, 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:13.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:19:13.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:19:13.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.120336295999998, 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:13.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.120064270999997, 2024-09-13T05:19:13.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:13.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:13.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:13.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.725 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.120020685, 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:13.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.731 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.119988458000002, 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:13.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:13.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.737 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.119777310000003, 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:13.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:13.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.119741017000003, 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:13.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:13.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:13.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.119702566999997, 2024-09-13T05:19:13.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:13.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:13.755 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.119527665, 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:13.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:13.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:13.761 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:13.763 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:13.763 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:13.762+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:13.864 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:13.862+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:13.935 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:13.935 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:13.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:13.964 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:13.966+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:14.065 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:14.058+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:14.068 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:14.068 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:14.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:14.166 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:14.162+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:14.244 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.112370650999999, 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:19:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:19:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:19:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:19:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:19:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:14.249 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:14.258 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:14.266 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:14.262+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:14.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:14.390 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:14.390 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:14.390 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:14.390 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:14.404 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:14.409 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:14.406+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:14.468 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:14.466+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:14.513 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:14.526 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:14.526 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:14.526 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:14.526 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:14.526 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:14.529 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:14.530 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:14.540 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:14.546 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:14.546 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:14.564 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:14.565 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:14.565 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:14.569 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:14.566+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:14.585 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:14.585 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:14.590 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:14.670 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:14.670+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:14.703 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:14.704 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:14.718 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:14.802 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:14.802+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:14.872 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:14.870+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:14.879 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:14.879 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:14.893 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:14.982 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:14.982+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:15.056 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:15.056 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:15.070 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:15.074 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:15.074+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:15.175 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:15.174+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:15.230 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:15.230 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:15.244 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:15.289 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:15.286+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:15.377 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:15.374+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:15.379 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:15.379 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:15.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:15.432 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:19:15.479 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:15.478+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:15.561 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:15.561 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:15.580 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:15.578+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:15.581 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:15.680 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:15.674+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:15.697 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:15.697 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:15.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:15.781 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:15.778+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:15.880 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:15.880 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:15.882 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:15.878+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:15.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:15.983 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:15.982+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:16.066 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:16.066 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:16.080 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:16.125 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:16.122+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:16.185 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:16.182+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:16.199 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:16.199 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:16.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:16.286 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:16.286+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:16.386 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:16.386 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:16.387 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:16.386+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:16.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:16.425 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:19:16.493 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:16.490+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:16.516 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 5, 3] 2024-09-13T05:19:16.516 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:19:16.516 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-13T05:19:16.516 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T05:19:16.516 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T05:19:16.517 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-13T05:19:16.523 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:16.523 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:16.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:16.589 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:16.586+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:16.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:16.649 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:16.649 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T05:19:16.649 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:16.689 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:16.686+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:16.707 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:16.707 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:16.723 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:16.765 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:16.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:16.781 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:16.790 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:16.786+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:16.797 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:16.797 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:16.814 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:16.815 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:16.815 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:16.833 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:16.833 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:16.837 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:16.848 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:16.848 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:16.848 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:16.848 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:16.862 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:16.894 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:16.890+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:16.977 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:16.977 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:16.977 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:16.977 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:16.977 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:16.991 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:16.994 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:16.990+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:17.095 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:17.094+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:17.117 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:17.117 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:17.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:17.117 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'53)", 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 67.155118324, 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050448339999999998, 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961434+0000", 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961433+0000", 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961492+0000", 2024-09-13T05:19:17.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9488999999999999e-05 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961517+0000", 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5296999999999999e-05 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961529+0000", 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1763000000000001e-05 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961621+0000", 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1636000000000002e-05 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.961656+0000", 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5363e-05 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966359+0000", 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047030839999999997 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:09.966479+0000", 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119839 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:17.122 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.984026568000004, 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:17.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:17.126 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.983395309000002, 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:17.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:17.131 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.982750187999997, 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:17.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:17.135 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.982242825, 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.136 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:17.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:17.140 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.981717388999996, 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.141 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.144 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.981079876999999, 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:17.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.980503169000002, 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:17.149 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.152 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.979951696000001, 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.154 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.979491112999995, 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:17.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.978981227000006, 2024-09-13T05:19:17.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.166 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.978554164000002, 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:17.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:17.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:116 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'78, mlcod=14'65)", 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.978093477000002, 2024-09-13T05:19:17.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050255040000000001, 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138459+0000", 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138458+0000", 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138469+0000", 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1596e-05 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138490+0000", 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0446e-05 2024-09-13T05:19:17.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138521+0000", 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0843000000000003e-05 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142091+0000", 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035703929999999998 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142146+0000", 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4873000000000001e-05 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143395+0000", 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012491169999999999 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143485+0000", 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9419000000000006e-05 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:117 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'79, mlcod=14'65)", 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.977601768, 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050956009999999999, 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:19:17.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138951+0000", 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138949+0000", 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138962+0000", 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2876e-05 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138987+0000", 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4652999999999999e-05 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139021+0000", 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4122999999999998e-05 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142737+0000", 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003715472 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.142773+0000", 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6086e-05 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144004+0000", 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001231631 2024-09-13T05:19:17.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.144046+0000", 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1996000000000002e-05 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:132 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'90, mlcod=14'86)", 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.594414502000006, 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005226835, 2024-09-13T05:19:17.180 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522138+0000", 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522137+0000", 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522142+0000", 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7040000000000002e-06 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522153+0000", 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0573e-05 2024-09-13T05:19:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522158+0000", 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9389999999999996e-06 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522191+0000", 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3571000000000001e-05 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.522204+0000", 2024-09-13T05:19:17.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3169999999999999e-05 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527326+0000", 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00512153 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.527365+0000", 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8971999999999999e-05 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.591794445999994, 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:17.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:17.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:153 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'107, mlcod=14'105)", 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 65.975467745000003, 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046950569999999999, 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.189 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141085+0000", 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141083+0000", 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141104+0000", 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0945999999999999e-05 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141127+0000", 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3054e-05 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.191 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141149+0000", 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1690000000000001e-05 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141266+0000", 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000117359 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.141303+0000", 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7013e-05 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145722+0000", 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044190260000000004 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.145780+0000", 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7751999999999999e-05 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 65.974259634000006, 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.194 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:17.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:156 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'110, mlcod=14'105)", 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 65.973680762000001, 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047886200000000004, 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142872+0000", 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142871+0000", 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:17.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142890+0000", 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9882999999999999e-05 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142944+0000", 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3465999999999999e-05 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142959+0000", 2024-09-13T05:19:17.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4644000000000001e-05 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144037+0000", 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001077988 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144094+0000", 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7079e-05 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147601+0000", 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035074400000000001 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147660+0000", 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9243000000000002e-05 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:159 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'113, mlcod=14'105)", 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 65.971995661999998, 2024-09-13T05:19:17.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044745929999999998, 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144557+0000", 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144556+0000", 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144583+0000", 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7416e-05 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144632+0000", 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9453999999999998e-05 2024-09-13T05:19:17.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.144651+0000", 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8635999999999999e-05 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.146970+0000", 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023192159999999998 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:17.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147003+0000", 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2595999999999998e-05 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.148955+0000", 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001951595 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.149031+0000", 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6922999999999995e-05 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.206 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.207 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.207 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:17.208 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:17.209 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:17.206+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:17.309 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:17.306+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:17.334 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:17.334 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:17.334 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:17.334 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:17.348 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:17.410 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:17.406+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:17.469 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:17.470 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:17.470 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:17.470 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:17.470 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:17.484 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:17.527 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:17.526+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:17.612 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:17.610+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:17.647 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:17.647 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:17.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4201.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 13.552029613, 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00040690000000000002, 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4201", 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.79:0/17281544", 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 2 2024-09-13T05:19:17.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:17.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097544+0000", 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097558+0000", 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3991999999999999e-05 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097581+0000", 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3238e-05 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097598+0000", 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7470999999999999e-05 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097717+0000", 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000118991 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097953+0000", 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023558699999999999 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.651 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.652 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.652 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:17.652 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:17.652 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:17.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:17.712 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:17.706+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:17.794 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:17.795 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:17.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:17.853 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:17.850+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:17.914 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:17.910+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:17.981 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:17.981 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:17.982 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:17.982 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:17.982 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:17.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:18.038 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:18.034+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:18.116 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:18.110+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:18.160 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 62.022070702000001, 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:18.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:19:18.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:19:18.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 62.021430733000003, 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015705100000000001, 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:18.165 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138449+0000", 2024-09-13T05:19:18.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138448+0000", 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138459+0000", 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0200000000000001e-05 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138474+0000", 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5438e-05 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138483+0000", 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5499999999999995e-06 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138523+0000", 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0831000000000003e-05 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138607+0000", 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3144000000000002e-05 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 61.026557672000003, 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:19:18.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:19:18.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:19:18.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.175 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 61.020445879999997, 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:18.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:19:18.179 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.016776141999998, 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:18.180 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:19:18.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:19:18.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:7 2.2 e19/16 ondisk, result = 0)", 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.010638423000003, 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000321014, 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149242+0000", 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149240+0000", 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149250+0000", 2024-09-13T05:19:18.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7990000000000005e-06 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149275+0000", 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5259999999999999e-05 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149289+0000", 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3627e-05 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149382+0000", 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3315999999999999e-05 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149394+0000", 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2216e-05 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149563+0000", 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168369 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.190 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4201.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:04.092330+0000", 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.067550365000001, 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00031489000000000002, 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4201", 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/17281544", 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:19:18.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092330+0000", 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092330+0000", 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092328+0000", 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092342+0000", 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4596000000000001e-05 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092364+0000", 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2229e-05 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092383+0000", 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8692e-05 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092473+0000", 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9740999999999993e-05 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:04.092645+0000", 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017189199999999999 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.194 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.195 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.195 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:18.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:18.237 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:18.234+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:18.318 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:18.314+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:18.366 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:18.366 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:18.366 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:18.366 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:18.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:18.421 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:18.418+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:18.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:19:18.520 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:18.518+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:18.550 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:18.550 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:18.550 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:18.550 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:18.550 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:18.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:18.620 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:18.614+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.832950639000003, 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010082459, 2024-09-13T05:19:18.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858218+0000", 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858216+0000", 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858229+0000", 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.305e-05 2024-09-13T05:19:18.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858266+0000", 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7523e-05 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858283+0000", 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6597000000000001e-05 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865356+0000", 2024-09-13T05:19:18.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070732859999999998 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865490+0000", 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133442 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.865698+0000", 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208202 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867616+0000", 2024-09-13T05:19:18.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019181409999999999 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868169+0000", 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055310299999999999 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868246+0000", 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6866000000000006e-05 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868300+0000", 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3912999999999997e-05 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.832862433000003, 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010336577, 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858306+0000", 2024-09-13T05:19:18.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858304+0000", 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858318+0000", 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3963e-05 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858337+0000", 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8482999999999999e-05 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.858351+0000", 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4195999999999999e-05 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866611+0000", 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082601199999999993 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866763+0000", 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015145600000000001 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.866987+0000", 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224441 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.867784+0000", 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079658099999999998 2024-09-13T05:19:18.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868531+0000", 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074704700000000003 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868596+0000", 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4966000000000001e-05 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.868642+0000", 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6607000000000003e-05 2024-09-13T05:19:18.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.738963870999996, 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:19:18.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:19:18.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.738756713000001, 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:19:18.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:19:18.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.718 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.561517984000005, 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010071422, 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 107 2024-09-13T05:19:18.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129650+0000", 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129649+0000", 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129656+0000", 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129668+0000", 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2463000000000001e-05 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.129675+0000", 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1999999999999997e-06 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131247+0000", 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015719359999999999 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131321+0000", 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4232000000000003e-05 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.131413+0000", 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1525000000000002e-05 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.133468+0000", 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002055177 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139645+0000", 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061772490000000001 2024-09-13T05:19:18.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139691+0000", 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5967000000000002e-05 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.139722+0000", 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0273e-05 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.724 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.560773759, 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:19:18.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:19:18.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:19:18.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.560644886999995, 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:19:18.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:19:18.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:19:18.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:19:18.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:19:18.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.560359594999994, 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010554714999999999, 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 115 2024-09-13T05:19:18.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130809+0000", 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130808+0000", 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130882+0000", 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4449999999999994e-05 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130891+0000", 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7190000000000007e-06 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130897+0000", 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135420+0000", 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045232060000000001 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135472+0000", 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1530999999999998e-05 2024-09-13T05:19:18.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135557+0000", 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5192999999999996e-05 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136761+0000", 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001204072 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141283+0000", 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045216099999999997 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141331+0000", 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8094999999999999e-05 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141363+0000", 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2319e-05 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.560265328, 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010559903000000001, 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 116 2024-09-13T05:19:18.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130903+0000", 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130902+0000", 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130905+0000", 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130909+0000", 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7239999999999998e-06 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130914+0000", 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1800000000000004e-06 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135956+0000", 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050418850000000003 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136008+0000", 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1694999999999999e-05 2024-09-13T05:19:18.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136090+0000", 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2527000000000005e-05 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137204+0000", 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011136640000000001 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141406+0000", 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042020039999999996 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141438+0000", 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2277999999999998e-05 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141463+0000", 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4396000000000002e-05 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.560247473000004, 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.751 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:19:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.175867150000002, 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:18.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:19:18.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.762 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.175830013999999, 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:18.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:19:18.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:19:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:18.768 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.175724095000007, 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:18.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:18.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:18.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.175452070000006, 2024-09-13T05:19:18.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:18.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.175408484000002, 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:18.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:18.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.175376256999996, 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:18.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.175165109000005, 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:18.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:18.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.175128815999997, 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.803 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.175090366000006, 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:18.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.174915463999994, 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.810 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:18.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:18.815 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:18.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:18.818 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:18.814+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:18.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:18.918 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:18.914+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:18.994 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:18.994 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:18.994 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:18.994 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:19.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:19.020 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:19.021 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:19.021 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:19.014+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:19.038 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:19.054 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:19.055 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:19.074 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:19.095 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:19.099 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:19.137 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:19.137 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:19.137 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:19.130+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:19.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:19.237 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:19.234+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:19.319 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:19.319 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:19.321 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:19.318+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:19.333 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:19.422 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:19.418+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:19.439 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:19:19.464 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:19.465 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:19.480 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:19.523 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:19.522+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:19.560 INFO:tasks.workunit.client.0.smithi122.stderr:Note: switching to '7a853b1706d076d58e0da7a5d3b9b9eb36276880'. 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr:state without impacting any branches by switching back to a branch. 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr: git switch -c 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr:Or undo this operation with: 2024-09-13T05:19:19.561 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-09-13T05:19:19.562 INFO:tasks.workunit.client.0.smithi122.stderr: git switch - 2024-09-13T05:19:19.562 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-09-13T05:19:19.562 INFO:tasks.workunit.client.0.smithi122.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-13T05:19:19.562 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-09-13T05:19:19.562 INFO:tasks.workunit.client.0.smithi122.stderr:HEAD is now at 7a853b1706d Merge pull request #59683 from zdover23/wip-doc-2024-09-10-backport-59677-to-reef 2024-09-13T05:19:19.568 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T05:19:19.616 INFO:tasks.workunit.client.0.smithi122.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T05:19:19.619 INFO:tasks.workunit.client.0.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T05:19:19.619 INFO:tasks.workunit.client.0.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T05:19:19.624 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:19.622+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:19.641 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:19.641 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:19.656 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:19.725 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:19.718+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:19.824 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:19.824 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:19.826 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:19.822+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:19.839 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:19.927 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:19.926+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:20.003 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:20.004 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:20.017 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:20.027 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:20.022+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:20.128 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:20.126+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:20.182 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:20.182 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:20.197 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:20.239 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:20.238+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:20.330 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:20.326+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:20.357 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:20.357 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:20.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:20.405 INFO:tasks.workunit.client.0.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T05:19:20.430 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:20.430+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:20.488 INFO:tasks.workunit.client.0.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T05:19:20.511 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:20.511 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:20.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:20.550 INFO:tasks.workunit.client.0.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T05:19:20.552 INFO:tasks.workunit.client.0.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T05:19:20.552 INFO:tasks.workunit.client.0.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T05:19:20.574 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:20.570+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:20.597 INFO:tasks.workunit.client.0.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T05:19:20.601 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T05:19:20.601 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T05:19:20.632 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:20.626+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:20.648 INFO:tasks.workunit:Running workunits matching rbd/rbd-nbd.sh on client.0... 2024-09-13T05:19:20.649 INFO:tasks.workunit:Running workunit rbd/rbd-nbd.sh... 2024-09-13T05:19:20.649 DEBUG:teuthology.orchestra.run.smithi122:workunit test rbd/rbd-nbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a853b1706d076d58e0da7a5d3b9b9eb36276880 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-13T05:19:20.650 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:20.650 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:20.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:20.700 INFO:tasks.workunit.client.0.smithi122.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-13T05:19:20.701 INFO:tasks.workunit.client.0.smithi122.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-09-13T05:19:20.701 INFO:tasks.workunit.client.0.smithi122.stderr:++ TIMEOUT=300 2024-09-13T05:19:20.702 INFO:tasks.workunit.client.0.smithi122.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-13T05:19:20.702 INFO:tasks.workunit.client.0.smithi122.stderr:++ MAX_TIMEOUT=15 2024-09-13T05:19:20.702 INFO:tasks.workunit.client.0.smithi122.stderr:++ PG_NUM=4 2024-09-13T05:19:20.702 INFO:tasks.workunit.client.0.smithi122.stderr:++ TMPDIR=/tmp 2024-09-13T05:19:20.702 INFO:tasks.workunit.client.0.smithi122.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-13T05:19:20.702 INFO:tasks.workunit.client.0.smithi122.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-13T05:19:20.702 INFO:tasks.workunit.client.0.smithi122.stderr:++ type xmlstarlet 2024-09-13T05:19:20.702 INFO:tasks.workunit.client.0.smithi122.stderr:++ XMLSTARLET=xmlstarlet 2024-09-13T05:19:20.702 INFO:tasks.workunit.client.0.smithi122.stderr:+++ uname 2024-09-13T05:19:20.703 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' Linux = FreeBSD ']' 2024-09-13T05:19:20.703 INFO:tasks.workunit.client.0.smithi122.stderr:++ SED=sed 2024-09-13T05:19:20.703 INFO:tasks.workunit.client.0.smithi122.stderr:++ AWK=awk 2024-09-13T05:19:20.704 INFO:tasks.workunit.client.0.smithi122.stderr:+++ stty -a 2024-09-13T05:19:20.704 INFO:tasks.workunit.client.0.smithi122.stderr:+++ head -1 2024-09-13T05:19:20.704 INFO:tasks.workunit.client.0.smithi122.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-13T05:19:20.726 INFO:tasks.workunit.client.0.smithi122.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T05:19:20.727 INFO:tasks.workunit.client.0.smithi122.stderr:++ termwidth= 2024-09-13T05:19:20.727 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-13T05:19:20.727 INFO:tasks.workunit.client.0.smithi122.stderr:++ DIFFCOLOPTS='-y ' 2024-09-13T05:19:20.727 INFO:tasks.workunit.client.0.smithi122.stderr:++ KERNCORE=kernel.core_pattern 2024-09-13T05:19:20.727 INFO:tasks.workunit.client.0.smithi122.stderr:++ EXTRA_OPTS= 2024-09-13T05:19:20.731 INFO:tasks.workunit.client.0.smithi122.stderr:++ test '' = TESTS 2024-09-13T05:19:20.731 INFO:tasks.workunit.client.0.smithi122.stderr:+ POOL=rbd 2024-09-13T05:19:20.731 INFO:tasks.workunit.client.0.smithi122.stderr:+ ANOTHER_POOL=new_default_pool21435 2024-09-13T05:19:20.731 INFO:tasks.workunit.client.0.smithi122.stderr:+ NS=ns 2024-09-13T05:19:20.731 INFO:tasks.workunit.client.0.smithi122.stderr:+ IMAGE=testrbdnbd21435 2024-09-13T05:19:20.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ SIZE=64 2024-09-13T05:19:20.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ DATA= 2024-09-13T05:19:20.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:20.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ setup 2024-09-13T05:19:20.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns x 2024-09-13T05:19:20.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -e CMakeCache.txt ']' 2024-09-13T05:19:20.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo echo test sudo 2024-09-13T05:19:20.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:20.732 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:20.733 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:20.733 INFO:tasks.workunit.client.0.smithi122.stderr:++ which echo 2024-09-13T05:19:20.734 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/echo 2024-09-13T05:19:20.734 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:20.734 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/echo test sudo 2024-09-13T05:19:20.742 INFO:tasks.workunit.client.0.smithi122.stdout:test sudo 2024-09-13T05:19:20.742 INFO:tasks.workunit.client.0.smithi122.stderr:+ trap cleanup INT TERM EXIT 2024-09-13T05:19:20.743 INFO:tasks.workunit.client.0.smithi122.stderr:++ mktemp -d 2024-09-13T05:19:20.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ TEMPDIR=/tmp/tmp.WvgaeitFGU 2024-09-13T05:19:20.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ DATA=/tmp/tmp.WvgaeitFGU/data 2024-09-13T05:19:20.744 INFO:tasks.workunit.client.0.smithi122.stderr:+ dd if=/dev/urandom of=/tmp/tmp.WvgaeitFGU/data bs=1M count=64 2024-09-13T05:19:20.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:20.746+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:20.826 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:20.826 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:20.834 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:20.830+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:20.840 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:20.934 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:20.930+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:20.958 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-09-13T05:19:20.958 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-09-13T05:19:20.959 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.213586 s, 314 MB/s 2024-09-13T05:19:20.959 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd namespace create rbd/ns 2024-09-13T05:19:21.006 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:21.006 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:21.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:21.033 INFO:tasks.workunit.client.0.smithi122.stderr:+ for ns in '' ${NS} 2024-09-13T05:19:21.033 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd --dest-pool rbd --dest-namespace '' --no-progress import /tmp/tmp.WvgaeitFGU/data testrbdnbd21435 2024-09-13T05:19:21.035 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:21.034+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:21.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:21.158 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:21.154+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:21.222 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:19:21.220+0000 7f2ad5cfd640 0 -- 172.21.15.122:0/3538759570 >> [v2:172.21.15.16:6808/2608485151,v1:172.21.15.16:6809/2608485151] conn(0x55fe46acf4b0 msgr2=0x55fe46ad18a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T05:19:21.231 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:21.232 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:21.234 INFO:tasks.workunit.client.0.smithi122.stderr:+ for ns in '' ${NS} 2024-09-13T05:19:21.234 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd --dest-pool rbd --dest-namespace ns --no-progress import /tmp/tmp.WvgaeitFGU/data testrbdnbd21435 2024-09-13T05:19:21.238 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:21.234+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:21.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:21.280 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:21.297 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:21.314 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:21.315 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:21.339 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:21.334+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:21.339 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:21.339 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:21.339 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:21.372 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:21.372 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:21.380 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.459055258999996, 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:19:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.458848101000001, 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:19:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:19:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.280865147, 2024-09-13T05:19:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010725095, 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 113 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130394+0000", 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130399+0000", 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1399999999999999e-06 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130405+0000", 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2029999999999997e-06 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130409+0000", 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134554+0000", 2024-09-13T05:19:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041455880000000004 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134606+0000", 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2046000000000003e-05 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.134694+0000", 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7676999999999994e-05 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136393+0000", 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016994709999999999 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141060+0000", 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046664929999999999 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141096+0000", 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6007999999999998e-05 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141120+0000", 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3669999999999999e-05 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.280736274999995, 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010712843, 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 114 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130523+0000", 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130527+0000", 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4229999999999998e-06 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130535+0000", 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3350000000000007e-06 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130540+0000", 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0969999999999999e-06 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135023+0000", 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044829689999999998 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135076+0000", 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3773000000000002e-05 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.135157+0000", 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0972999999999999e-05 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136401+0000", 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012434410000000001 2024-09-13T05:19:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141167+0000", 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047667940000000004 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141212+0000", 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4441999999999999e-05 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141236+0000", 2024-09-13T05:19:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4219e-05 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.280338861000004, 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:19:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:19:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895958538000002, 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:19:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:19:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895921401999999, 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:19:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:19:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:19:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895815483000007, 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895543458000006, 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895499872000002, 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895467644999997, 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895256497000005, 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895220203999997, 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895181754000006, 2024-09-13T05:19:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.895006851999995, 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.080302914000000003, 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:19:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.072772800999999998, 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:19:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.063203473999999996, 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:19:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:19:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.052040601999999998, 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:19:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.045881005000000002, 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:19:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:19:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:19:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.533 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:21.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:19:21.535 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:21.534+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:21.535 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:19:21.440+0000 7f2734a08640 0 -- 172.21.15.122:0/106628764 >> [v2:172.21.15.16:6800/2153409514,v1:172.21.15.16:6801/2153409514] conn(0x55948cbd61e0 msgr2=0x55948cbd85d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T05:19:21.535 INFO:tasks.workunit.client.0.smithi122.stderr:+ ceph osd pool create new_default_pool21435 8 2024-09-13T05:19:21.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:21.635 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:21.630+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:21.664 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 5, 3] 2024-09-13T05:19:21.664 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:19:21.664 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T05:19:21.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T05:19:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:21.702 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:21.702 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:21.703 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:21.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:21.762 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:21.758+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:21.837 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:21.834+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:21.880 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:21.880 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:21.880 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:21.880 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:21.880 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:21.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:21.942 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:21.938+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:21.967 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:19:21.967 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 46, 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5", 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2024-09-13T05:17:57.363809+0000", 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2024-09-13T05:19:19.427718+0000", 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2024-09-13T05:19:05.362831+0000", 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2024-09-13T05:17:59.738330+0000", 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5804032, 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "nodeep-scrub", 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2024-09-13T05:19:21.968 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 6, 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "jewel", 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "reef", 2024-09-13T05:19:21.969 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_crimson": false, 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": ".mgr", 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:08.122164+0000", 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:19:21.970 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:19:21.971 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "15", 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:19:21.972 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:19:21.973 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:19:21.974 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": {} 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 6.059999942779541, 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 6.059999942779541, 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.33000001311302185, 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2, 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2, 2024-09-13T05:19:21.975 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:15.001664+0000", 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:19:21.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:19:21.977 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "19", 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 3, 2024-09-13T05:19:21.978 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 19, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:19:21.979 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:19:21.980 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 2.2699999809265137, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 2.2699999809265137, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.82999998331069946, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-13T05:19:21.981 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "e58174f4-85ec-4321-8bc3-384261c59c6b", 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:21.982 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 40, 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/2153409514", 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/2153409514", 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/2153409514", 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/2153409514", 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:21.987 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "84a3db6c-dacd-43d5-b964-37fdef0091a4", 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:21.988 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 40, 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:21.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:21.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.991 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.991 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:21.991 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.991 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:21.992 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/2608485151", 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/2608485151", 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/2608485151", 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/2608485151", 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.994 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "0d530476-cf38-4544-9ce7-e8ed35df220b", 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 16, 2024-09-13T05:19:21.995 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:21.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2024-09-13T05:19:21.999 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/1006404377", 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/1006404377", 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/1006404377", 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/1006404377", 2024-09-13T05:19:22.000 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "fd91f7e2-580d-47ea-9db5-792c8f1ebe01", 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.980804443359375, 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 20, 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 29, 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 29, 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 21, 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.002 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6816", 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6817", 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.003 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6818", 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6819", 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.004 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6822", 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6823", 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6820", 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6821", 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6817/1834661236", 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6819/1834661236", 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6823/1834661236", 2024-09-13T05:19:22.006 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6821/1834661236", 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "01af0f4c-5da2-46fe-a262-59fcc6a2efa9", 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 0, 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:22.007 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 16, 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 40, 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6808", 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6809", 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6810", 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:19:22.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6811", 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6814", 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:19:22.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6815", 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.011 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6812", 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6813", 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2630134185 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6809/2630134185", 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6811/2630134185", 2024-09-13T05:19:22.012 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6815/2630134185", 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6813/2630134185", 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: "exists" 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "64cdd5f2-1568-4e96-ad90-6ef7414651f0", 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:22.013 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 29, 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.014 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6800", 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6801", 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6802", 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6803", 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6806", 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6807", 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:22.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6804", 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6805", 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6801/2778785304", 2024-09-13T05:19:22.018 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6803/2778785304", 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6807/2778785304", 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6805/2778785304", 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:22.019 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.193869+0000", 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:22.020 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.255764+0000", 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.286354+0000", 2024-09-13T05:19:22.021 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.221310+0000", 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 21 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.022 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.026235+0000", 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 40 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:22.023 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.010242+0000", 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [], 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_primaries": [], 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2024-09-13T05:19:22.024 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2024-09-13T05:19:22.025 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2024-09-13T05:19:22.026 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2024-09-13T05:19:22.026 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2024-09-13T05:19:22.026 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2024-09-13T05:19:22.026 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2024-09-13T05:19:22.026 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2024-09-13T05:19:22.026 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2024-09-13T05:19:22.026 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2024-09-13T05:19:22.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.026 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:22.028 INFO:tasks.thrashosds.thrasher:j is {'epoch': 46, 'fsid': '1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5', 'created': '2024-09-13T05:17:57.363809+0000', 'modified': '2024-09-13T05:19:19.427718+0000', 'last_up_change': '2024-09-13T05:19:05.362831+0000', 'last_in_change': '2024-09-13T05:17:59.738330+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T05:18:08.122164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T05:18:15.001664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e58174f4-85ec-4321-8bc3-384261c59c6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 2153409514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 2153409514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 2153409514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 2153409514}]}, 'public_addr': '172.21.15.16:6801/2153409514', 'cluster_addr': '172.21.15.16:6803/2153409514', 'heartbeat_back_addr': '172.21.15.16:6807/2153409514', 'heartbeat_front_addr': '172.21.15.16:6805/2153409514', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '84a3db6c-dacd-43d5-b964-37fdef0091a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 2608485151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 2608485151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 2608485151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 2608485151}]}, 'public_addr': '172.21.15.16:6809/2608485151', 'cluster_addr': '172.21.15.16:6811/2608485151', 'heartbeat_back_addr': '172.21.15.16:6815/2608485151', 'heartbeat_front_addr': '172.21.15.16:6813/2608485151', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0d530476-cf38-4544-9ce7-e8ed35df220b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 1006404377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 1006404377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 1006404377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 1006404377}]}, 'public_addr': '172.21.15.16:6817/1006404377', 'cluster_addr': '172.21.15.16:6819/1006404377', 'heartbeat_back_addr': '172.21.15.16:6823/1006404377', 'heartbeat_front_addr': '172.21.15.16:6821/1006404377', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fd91f7e2-580d-47ea-9db5-792c8f1ebe01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.980804443359375, 'last_clean_begin': 12, 'last_clean_end': 20, 'up_from': 29, 'up_thru': 29, 'down_at': 21, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6816', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6817', 'nonce': 1834661236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6818', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6819', 'nonce': 1834661236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6822', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6823', 'nonce': 1834661236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6820', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6821', 'nonce': 1834661236}]}, 'public_addr': '172.21.15.79:6817/1834661236', 'cluster_addr': '172.21.15.79:6819/1834661236', 'heartbeat_back_addr': '172.21.15.79:6823/1834661236', 'heartbeat_front_addr': '172.21.15.79:6821/1834661236', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '01af0f4c-5da2-46fe-a262-59fcc6a2efa9', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 16, 'down_at': 40, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6808', 'nonce': 2630134185}, {'type': 'v1', 'addr': '172.21.15.79:6809', 'nonce': 2630134185}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6810', 'nonce': 2630134185}, {'type': 'v1', 'addr': '172.21.15.79:6811', 'nonce': 2630134185}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6814', 'nonce': 2630134185}, {'type': 'v1', 'addr': '172.21.15.79:6815', 'nonce': 2630134185}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6812', 'nonce': 2630134185}, {'type': 'v1', 'addr': '172.21.15.79:6813', 'nonce': 2630134185}]}, 'public_addr': '172.21.15.79:6809/2630134185', 'cluster_addr': '172.21.15.79:6811/2630134185', 'heartbeat_back_addr': '172.21.15.79:6815/2630134185', 'heartbeat_front_addr': '172.21.15.79:6813/2630134185', 'state': ['exists']}, {'osd': 5, 'uuid': '64cdd5f2-1568-4e96-ad90-6ef7414651f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6800', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6801', 'nonce': 2778785304}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6802', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6803', 'nonce': 2778785304}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6806', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6807', 'nonce': 2778785304}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6804', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6805', 'nonce': 2778785304}]}, 'public_addr': '172.21.15.79:6801/2778785304', 'cluster_addr': '172.21.15.79:6803/2778785304', 'heartbeat_back_addr': '172.21.15.79:6807/2778785304', 'heartbeat_front_addr': '172.21.15.79:6805/2778785304', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.193869+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.255764+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.286354+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.221310+0000', 'dead_epoch': 21}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.026235+0000', 'dead_epoch': 40}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.010242+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T05:19:22.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:19:22.061 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.927997130999998, 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:22.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.066 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.73249199700000001, 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:19:22.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:22.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.072 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.72128623599999997, 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:19:22.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.711107182, 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:22.077 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:22.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:19:22.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4571.0:35 2.0 e46/16 ondisk, result = 0)", 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.355775+0000", 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.704372832, 2024-09-13T05:19:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026547000000000002, 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355775+0000", 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355775+0000", 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355774+0000", 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355784+0000", 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0515e-05 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355810+0000", 2024-09-13T05:19:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5554e-05 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355821+0000", 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1252e-05 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355884+0000", 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3225999999999994e-05 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355898+0000", 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.343e-05 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356041+0000", 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014299000000000001 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.085 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.70367137899999999, 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:19:22.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:19:22.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:22.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.69706367599999997, 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:19:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.68783745799999996, 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:19:22.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4571.0:38 2.0 e46/16 ondisk, result = 0)", 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.384396+0000", 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.67575212399999995, 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015359, 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384396+0000", 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384396+0000", 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384395+0000", 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384403+0000", 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5860000000000005e-06 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384416+0000", 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2954e-05 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384424+0000", 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2024-09-13T05:19:22.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384461+0000", 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6863999999999998e-05 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384470+0000", 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8060000000000001e-06 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384549+0000", 2024-09-13T05:19:22.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9828999999999998e-05 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4571.0:46 2.0 e46/16 ondisk, result = 0)", 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.387863+0000", 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.67228474100000002, 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134473, 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:22.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387863+0000", 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387863+0000", 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387863+0000", 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:22.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387868+0000", 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5759999999999996e-06 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387878+0000", 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0395e-05 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387884+0000", 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4859999999999997e-06 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387917+0000", 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2975000000000002e-05 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387922+0000", 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3630000000000002e-06 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:22.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:22.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387998+0000", 2024-09-13T05:19:22.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5339999999999994e-05 2024-09-13T05:19:22.111 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.111 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.111 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.111 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:22.111 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:22.111 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:22.112 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:22.058+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:22.113 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:22.158 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:22.154+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:22.242 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:22.238+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:22.244 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:22.244 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:22.244 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:22.244 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:22.259 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:22.343 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:22.338+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:22.413 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:19:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:19:22.427 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:22.427 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:22.432 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-13T05:19:22.066413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4784707,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":101,"num_write":163,"num_write_kb":6515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":15,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":179496,"kb_used_data":4000,"kb_used_omap":12,"kb_used_meta":175411,"kb_avail":566051544,"statfs":{"total":579820584960,"available":579636781056,"internally_reserved":0,"allocated":4096000,"data_stored":2220489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12399,"internal_metadata":179621777},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4194320,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":2,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002102"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-13T05:19:13.411016+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:19:13.411016+0000","last_peered":"2024-09-13T05:19:13.411016+0000","last_clean":"2024-09-13T05:19:13.411016+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:19:13.411016+0000","last_undegraded":"2024-09-13T05:19:13.411016+0000","last_fullsized":"2024-09-13T05:19:13.411016+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":43,"state":"active+undersized","last_fresh":"2024-09-13T05:19:10.404748+0000","last_change":"2024-09-13T05:19:06.372827+0000","last_active":"2024-09-13T05:19:10.404748+0000","last_peered":"2024-09-13T05:19:10.404748+0000","last_clean":"2024-09-13T05:18:15.101421+0000","last_became_active":"2024-09-13T05:19:06.372827+0000","last_became_peered":"2024-09-13T05:19:06.372827+0000","last_unstale":"2024-09-13T05:19:10.404748+0000","last_undegraded":"2024-09-13T05:19:10.404748+0000","last_fullsized":"2024-09-13T05:19:06.371900+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:45:05.712505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-09-13T05:19:13.411036+0000","last_change":"2024-09-13T05:19:06.374860+0000","last_active":"2024-09-13T05:19:13.411036+0000","last_peered":"2024-09-13T05:19:13.411036+0000","last_clean":"2024-09-13T05:19:01.360503+0000","last_became_active":"2024-09-13T05:19:06.374860+0000","last_became_peered":"2024-09-13T05:19:06.374860+0000","last_unstale":"2024-09-13T05:19:13.411036+0000","last_undegraded":"2024-09-13T05:19:13.411036+0000","last_fullsized":"2024-09-13T05:19:06.373909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:11:52.912925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-13T05:19:10.407425+0000","last_change":"2024-09-13T05:18:43.275214+0000","last_active":"2024-09-13T05:19:10.407425+0000","last_peered":"2024-09-13T05:19:10.407425+0000","last_clean":"2024-09-13T05:19:10.407425+0000","last_became_active":"2024-09-13T05:18:43.274886+0000","last_became_peered":"2024-09-13T05:18:43.274886+0000","last_unstale":"2024-09-13T05:19:10.407425+0000","last_undegraded":"2024-09-13T05:19:10.407425+0000","last_fullsized":"2024-09-13T05:19:10.407425+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:53.129994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":47,"reported_epoch":43,"state":"active+undersized+degraded","last_fresh":"2024-09-13T05:19:10.404707+0000","last_change":"2024-09-13T05:19:06.372773+0000","last_active":"2024-09-13T05:19:10.404707+0000","last_peered":"2024-09-13T05:19:10.404707+0000","last_clean":"2024-09-13T05:19:02.230820+0000","last_became_active":"2024-09-13T05:19:06.372773+0000","last_became_peered":"2024-09-13T05:19:06.372773+0000","last_unstale":"2024-09-13T05:19:10.404707+0000","last_undegraded":"2024-09-13T05:19:06.371983+0000","last_fullsized":"2024-09-13T05:19:06.371900+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:52:22.160291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'2","reported_seq":48,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-13T05:19:21.300012+0000","last_change":"2024-09-13T05:18:43.272711+0000","last_active":"2024-09-13T05:19:21.300012+0000","last_peered":"2024-09-13T05:19:21.300012+0000","last_clean":"2024-09-13T05:19:21.300012+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:19:21.300012+0000","last_undegraded":"2024-09-13T05:19:21.300012+0000","last_fullsized":"2024-09-13T05:19:21.300012+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:27.589949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194320,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":2,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-13T05:19:13.410018+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:19:13.410018+0000","last_peered":"2024-09-13T05:19:13.410018+0000","last_clean":"2024-09-13T05:19:13.410018+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:19:13.410018+0000","last_undegraded":"2024-09-13T05:19:13.410018+0000","last_fullsized":"2024-09-13T05:19:13.410018+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":41,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-13T05:19:13.410995+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:19:13.410995+0000","last_peered":"2024-09-13T05:19:13.410995+0000","last_clean":"2024-09-13T05:19:13.410995+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:19:13.410995+0000","last_undegraded":"2024-09-13T05:19:13.410995+0000","last_fullsized":"2024-09-13T05:19:13.410995+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":188,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-13T05:19:13.410959+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:19:13.410959+0000","last_peered":"2024-09-13T05:19:13.410959+0000","last_clean":"2024-09-13T05:19:13.410959+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:19:13.410959+0000","last_undegraded":"2024-09-13T05:19:13.410959+0000","last_fullsized":"2024-09-13T05:19:13.410959+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194339,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":4,"num_write_kb":4099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12583004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607571,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343764,"statfs":{"total":96636764160,"available":96608014336,"internally_reserved":0,"allocated":1081344,"data_stored":765200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.623}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76500000000000001}]},{"osd":2,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":4,"up_from":12,"seq":51539607568,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":417792,"data_stored":147294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28608,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94343232,"statfs":{"total":96636764160,"available":96607469568,"internally_reserved":0,"allocated":512000,"data_stored":183972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":2,"up_from":12,"seq":51539607570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31604,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340236,"statfs":{"total":96636764160,"available":96604401664,"internally_reserved":0,"allocated":499712,"data_stored":179402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.35699999999999998}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.32900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":5,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.68700000000000006}]}]},{"osd":1,"up_from":12,"seq":51539607570,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1093632,"data_stored":769770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.379}]},{"osd":2,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.41199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":5,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.52000000000000002}]}]},{"osd":0,"up_from":12,"seq":51539607570,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":491520,"data_stored":174851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.66700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":5,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.70499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:19:22.434 INFO:tasks.thrashosds.thrasher:Setting 2.2 to [0, 1] 2024-09-13T05:19:22.434 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.2', '0', '1'] 2024-09-13T05:19:22.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.2 0 1 2024-09-13T05:19:22.442 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:22.443 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:22.442+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:22.447 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:19:22.544 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:22.538+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:22.544 INFO:tasks.workunit.client.0.smithi122.stderr:pool 'new_default_pool21435' already exists 2024-09-13T05:19:22.561 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd pool init new_default_pool21435 2024-09-13T05:19:22.567 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:22.567 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:22.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:22.567 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:22.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.567 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:22.567 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:22.567 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.433323016000003, 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:22.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.432691757000001, 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.572 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:22.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.432046635999995, 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:22.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:22.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.431539272999999, 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.581 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:22.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:22.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:22.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:22.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:22.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.431013836999995, 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.590 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.430376324999997, 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:22.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:22.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:22.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:22.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.429799617, 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:22.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:22.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.429248143999999, 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.600 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:22.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.602 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:22.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:22.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:22.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:22.604 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.604 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.428787560999993, 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.606 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:22.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:22.609 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.428277675000004, 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:22.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.611 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:22.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.427850612, 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:22.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:22.618 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.041090894000007, 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:22.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:22.622 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 71.423556082000005, 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.624 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:22.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.1927771469999999, 2024-09-13T05:19:22.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:19:22.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.631 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.182156803, 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.632 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:19:22.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.637 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.174942489, 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:19:22.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:22.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.643 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.1685869689999999, 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:19:22.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:22.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.649 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.1685306049999999, 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:19:22.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:19:22.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.1684930339999999, 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:22.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:19:22.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:19:22.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.16845851, 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:22.662 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:22.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:22.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:19:22.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.667 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:22.668 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:22.668 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:22.669 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:22.669 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:22.666+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:22.770 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:22.766+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:22.840 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:22.840 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:22.840 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:22.840 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:22.854 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:22.871 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:22.866+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:22.970 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:22.970 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:22.970 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:22.970 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:22.970 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:22.971 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:22.966+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:22.985 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:23.072 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:23.066+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:23.113 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:23.113 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4201.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 19.015751627, 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00040690000000000002, 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4201", 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.79:0/17281544", 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 2 2024-09-13T05:19:23.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097546+0000", 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097544+0000", 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097558+0000", 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3991999999999999e-05 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097581+0000", 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3238e-05 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097598+0000", 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7470999999999999e-05 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097717+0000", 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000118991 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:04.097953+0000", 2024-09-13T05:19:23.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023558699999999999 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.7724433849999999, 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:23.118 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:19:23.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:19:23.123 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.7605627830000001, 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:23.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:19:23.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:19:23.127 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.754619084, 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:23.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.7519433760000001, 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.132 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:23.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:19:23.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:19:23.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.7472293429999999, 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:19:23.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop_reply(client.4571.0:40 2.1 e46/29 ondisk, result = 0)", 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.377410+0000", 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.735887478, 2024-09-13T05:19:23.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00043392399999999997, 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377410+0000", 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377410+0000", 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.141 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377406+0000", 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377428+0000", 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1285999999999999e-05 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377464+0000", 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6831e-05 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377488+0000", 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3728000000000001e-05 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377606+0000", 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011790000000000001 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377624+0000", 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7782e-05 2024-09-13T05:19:23.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377844+0000", 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022016700000000001 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:49 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'17, mlcod=46'15)", 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.379251+0000", 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.734046966, 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001378642, 2024-09-13T05:19:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379251+0000", 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379251+0000", 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379249+0000", 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379264+0000", 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5752000000000001e-05 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379291+0000", 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6375000000000001e-05 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379306+0000", 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.543e-05 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379411+0000", 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000104886 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379448+0000", 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6897999999999998e-05 2024-09-13T05:19:23.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.380537+0000", 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001088901 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.380629+0000", 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2287000000000006e-05 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.73127707, 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.149 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.152 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.721636868, 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:23.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.154 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.714544063, 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.157 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:19:23.161 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.706326107, 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:19:23.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:19:23.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.7041802939999999, 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:19:23.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:19:23.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.7022625, 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:19:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:56 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'24, mlcod=46'17)", 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.412806+0000", 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.7004917559999999, 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001612657, 2024-09-13T05:19:23.174 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412806+0000", 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412806+0000", 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412804+0000", 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412819+0000", 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4842e-05 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412847+0000", 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7458999999999999e-05 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412864+0000", 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7141000000000001e-05 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413020+0000", 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000156346 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413060+0000", 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9774999999999998e-05 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414357+0000", 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012974569999999999 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414418+0000", 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1062999999999995e-05 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.178 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:61 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'25, mlcod=46'23)", 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.440719+0000", 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.6725782890000001, 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001104063, 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.440719+0000", 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:23.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.440719+0000", 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.440717+0000", 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.440734+0000", 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6976000000000001e-05 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.440770+0000", 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6191e-05 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.440789+0000", 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8876e-05 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.440904+0000", 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011523600000000001 2024-09-13T05:19:23.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.440941+0000", 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6531e-05 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.441728+0000", 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078714599999999996 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.441823+0000", 2024-09-13T05:19:23.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5670999999999999e-05 2024-09-13T05:19:23.183 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.183 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.183 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.183 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:23.183 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:23.183 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:23.184 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:23.182+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:23.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:23.286 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:23.282+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:23.320 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:23.320 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:23.321 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:23.321 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:23.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:23.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:23.386 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:23.386+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.2 pg_upmap mapping to [0,1] 2024-09-13T05:19:23.461 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:23.461 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:23.462 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:23.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:23.488 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:23.482+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:23.533 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:23.533 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:23.549 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:23.549 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:23.566 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:23.566 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:23.586 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:23.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:23.586+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:23.606 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:23.606 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:23.606 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:23.610 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:23.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:23.709 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:23.706+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:23.790 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:23.786+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:23.791 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:23.791 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:23.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:23.891 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:23.890+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:23.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:23.992 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:23.986+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:24.093 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:24.086+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:24.161 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:24.162 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:24.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:24.218 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:24.214+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:24.295 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:24.290+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:24.336 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:24.336 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:24.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:24.396 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:24.390+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:24.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:19:24.514 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:24.510+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:24.527 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:24.527 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:24.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:24.597 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:24.598+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:24.698 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:24.694+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:24.720 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:24.721 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:24.735 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:24.799 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:24.794+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:24.865 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:24.865 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:24.880 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:24.927 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:24.926+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:25.001 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:25.001 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:25.002 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:24.998+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:25.015 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:25.107 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:25.106+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:25.185 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:25.185 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:25.199 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:25.243 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:25.242+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:25.304 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:25.302+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:25.364 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:25.364 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:25.378 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:25.406 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:25.402+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:25.474 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:19:25.489 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd-nbd 2024-09-13T05:19:25.489 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd-nbd 2024-09-13T05:19:25.506 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:25.502+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:25.521 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: must specify command 2024-09-13T05:19:25.523 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:25.523 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd-nbd INVALIDCMD 2024-09-13T05:19:25.523 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd-nbd INVALIDCMD 2024-09-13T05:19:25.546 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:25.546 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:25.548 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: unknown command: INVALIDCMD 2024-09-13T05:19:25.550 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:25.550 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:25.552 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -ne 0 ']' 2024-09-13T05:19:25.552 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd device --device-type nbd map testrbdnbd21435 2024-09-13T05:19:25.552 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd map testrbdnbd21435 2024-09-13T05:19:25.560 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:25.607 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:25.606+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:25.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:25.665 INFO:tasks.workunit.client.0.smithi122.stderr:modprobe: ERROR: could not insert 'nbd': Operation not permitted 2024-09-13T05:19:25.667 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: failed to find unused device 2024-09-13T05:19:25.677 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-13T05:19:25.681 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:25.681 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-13T05:19:25.681 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-13T05:19:25.681 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:25.681 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:25.682 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:25.682 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:25.683 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:25.683 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:25.683 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd map INVALIDIMAGE 2024-09-13T05:19:25.708 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:25.706+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:25.729 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:25.730 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:25.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:25.773 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-13T05:19:25.777 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:25.778 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd-nbd --device INVALIDDEV map testrbdnbd21435 2024-09-13T05:19:25.778 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd-nbd --device INVALIDDEV map testrbdnbd21435 2024-09-13T05:19:25.778 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:25.778 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:25.778 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:25.778 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd-nbd 2024-09-13T05:19:25.779 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-13T05:19:25.780 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:25.780 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd-nbd --device INVALIDDEV map testrbdnbd21435 2024-09-13T05:19:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:25.807 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:25.808 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:25.802+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:25.848 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: invalid device path: INVALIDDEV (expected /dev/nbd{num}) 2024-09-13T05:19:25.850 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:25.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:25.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:25.858 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:25.859 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd device --device-type nbd --format INVALID list 2024-09-13T05:19:25.859 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd --format INVALID list 2024-09-13T05:19:25.866 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:25.866 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:25.866 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:25.866 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:25.880 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:25.883 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: the argument for option '--format' is invalid 2024-09-13T05:19:25.885 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:25.885 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd --format json --pretty-format list 2024-09-13T05:19:25.909 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:25.910+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:25.934 INFO:tasks.workunit.client.0.smithi122.stdout:[] 2024-09-13T05:19:25.937 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd --format xml list 2024-09-13T05:19:25.990 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:25.990 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:25.990 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:25.991 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:25.991 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:25.993 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:25.993 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:25.993 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:26.009 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:26.002+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:26.045 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:26.045 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:26.045 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:26.045 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:26.045 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:26.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:26.106 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:26.106 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:26.106 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:26.106 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:26.107 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:26.107 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:26.110 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:26.106+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:26.182 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=21740 2024-09-13T05:19:26.183 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 21740 2024-09-13T05:19:26.183 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 21740 -C rbd-nbd 2024-09-13T05:19:26.190 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:26.190 INFO:tasks.workunit.client.0.smithi122.stdout: 21740 ? 00:00:00 rbd-nbd 2024-09-13T05:19:26.191 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21435 2024-09-13T05:19:26.191 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21435 2024-09-13T05:19:26.191 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:26.191 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:26.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:26.192 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd-nbd 2024-09-13T05:19:26.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-13T05:19:26.195 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:26.195 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21435 2024-09-13T05:19:26.211 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:26.210+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:26.234 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.093501246000002, 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000398942, 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:26.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137810+0000", 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137804+0000", 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999948 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137817+0000", 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2157999999999999e-05 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137831+0000", 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4744e-05 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137846+0000", 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4486e-05 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.137949+0000", 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010273899999999999 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.138208+0000", 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025995499999999997 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.238 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.097988216000005, 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:19:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:19:26.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:19:26.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4190.0:5 2.2 e18/16 ondisk, result = 0)", 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.091876424000006, 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040807300000000001, 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139434+0000", 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139433+0000", 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139444+0000", 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1483e-05 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139471+0000", 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6886999999999999e-05 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139498+0000", 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7183e-05 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139599+0000", 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132900000000001 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139617+0000", 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8017e-05 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139842+0000", 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224971 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.248 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.088206686000007, 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:19:26.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:19:26.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:19:26.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:6 2.6 2:774f7988:ns::rbd_directory:head [call rbd.dir_state_assert in=19b,call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.255 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.285876+0000", 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.9454351519999999, 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099486699999999997, 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 6 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.285876+0000", 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.285876+0000", 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.285875+0000", 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.285883+0000", 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5720000000000003e-06 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.285900+0000", 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6379e-05 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.285908+0000", 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8669999999999994e-06 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.285957+0000", 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9057000000000001e-05 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.286017+0000", 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0742000000000002e-05 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.286795+0000", 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00077729600000000002 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.286826+0000", 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1686999999999997e-05 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.286870+0000", 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4181000000000003e-05 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.260 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:8 2.6 2:774f7988:ns::rbd_directory:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.290496+0000", 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.9408150649999998, 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00035262899999999997, 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 8 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.290496+0000", 2024-09-13T05:19:26.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.290496+0000", 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.290494+0000", 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.290502+0000", 2024-09-13T05:19:26.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3590000000000002e-06 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.290516+0000", 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4447000000000001e-05 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.290523+0000", 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.759e-06 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.290580+0000", 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7046000000000002e-05 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.290688+0000", 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107812 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.290848+0000", 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160438 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:9 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.291157+0000", 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.9401541760000001, 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00096355799999999999, 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9 2024-09-13T05:19:26.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.291157+0000", 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.291157+0000", 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.291156+0000", 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.291161+0000", 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4940000000000003e-06 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.291173+0000", 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1484000000000001e-05 2024-09-13T05:19:26.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.291179+0000", 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7620000000000001e-06 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.291214+0000", 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5533e-05 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.291326+0000", 2024-09-13T05:19:26.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112229 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.292024+0000", 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069747600000000004 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.292059+0000", 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5689999999999999e-05 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.292120+0000", 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0690000000000003e-05 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:13 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.299725+0000", 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.931585858, 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00055138599999999996, 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 13 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.299725+0000", 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.299725+0000", 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.299723+0000", 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.299734+0000", 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0477e-05 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.299754+0000", 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0012000000000001e-05 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.299767+0000", 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3189e-05 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.299832+0000", 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.512e-05 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.299944+0000", 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112248 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.300276+0000", 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033221599999999998 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:17 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [call rbd.get_size in=11b,call rbd.get_features in=15b,call rbd.get_flags in=12b,call,call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.302175+0000", 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.9291354590000003, 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000388802, 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.302175+0000", 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.302175+0000", 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.302175+0000", 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.302181+0000", 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.601e-06 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.302195+0000", 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3562e-05 2024-09-13T05:19:26.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.302204+0000", 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.302245+0000", 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0531999999999998e-05 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.302297+0000", 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2187e-05 2024-09-13T05:19:26.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.302564+0000", 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026715199999999999 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:22 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [watch watch cookie 139805748911888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.304986+0000", 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.9263246130000002, 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099310000000000002, 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.304986+0000", 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.304986+0000", 2024-09-13T05:19:26.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.304985+0000", 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.304989+0000", 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.856e-06 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.304999+0000", 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9459999999999993e-06 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.305005+0000", 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3909999999999997e-06 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.305057+0000", 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2496e-05 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.305144+0000", 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6822000000000003e-05 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.305888+0000", 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074437899999999998 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.305917+0000", 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8952999999999999e-05 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.305979+0000", 2024-09-13T05:19:26.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1968e-05 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:24 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.311079+0000", 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.9202320569999998, 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00109673, 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.311079+0000", 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.311079+0000", 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.311078+0000", 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.311083+0000", 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6280000000000002e-06 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.311097+0000", 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3794999999999999e-05 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.311106+0000", 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5310000000000001e-06 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.311159+0000", 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3204000000000001e-05 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.311258+0000", 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9208000000000003e-05 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.312099+0000", 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00084089400000000002 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.312132+0000", 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2977e-05 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.312175+0000", 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3371000000000002e-05 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.287 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.8653764329999998, 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:26.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.8583468380000001, 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:19:26.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:26.297 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.8175640030000002, 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:19:26.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.8107055299999999, 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:19:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:62 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.440913+0000", 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7903975680000004, 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001022235, 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 62 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.440913+0000", 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.440913+0000", 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.440910+0000", 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.440926+0000", 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6082999999999998e-05 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.440958+0000", 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1828000000000001e-05 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.440991+0000", 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2796000000000002e-05 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.441045+0000", 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3745000000000001e-05 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.441116+0000", 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1155000000000006e-05 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.441850+0000", 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00073442699999999995 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.441896+0000", 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5512000000000001e-05 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.441935+0000", 2024-09-13T05:19:26.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9814999999999997e-05 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:64 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [watch unwatch cookie 139805748911888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7888683939999996, 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012844899999999999, 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 64 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442441+0000", 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442448+0000", 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6540000000000002e-06 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442465+0000", 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7164999999999999e-05 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442472+0000", 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2599999999999999e-06 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442630+0000", 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015837800000000001 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442736+0000", 2024-09-13T05:19:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000105449 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443603+0000", 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00086757000000000002 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443639+0000", 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5806999999999998e-05 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443727+0000", 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7646999999999996e-05 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:65 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.442860+0000", 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7884512350000001, 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099624299999999995, 2024-09-13T05:19:26.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 65 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442860+0000", 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442860+0000", 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442858+0000", 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442868+0000", 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9380000000000004e-06 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442888+0000", 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0539000000000001e-05 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442903+0000", 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4194e-05 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443265+0000", 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00036276800000000002 2024-09-13T05:19:26.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443321+0000", 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5884000000000002e-05 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443758+0000", 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00043674399999999998 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443798+0000", 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0114000000000003e-05 2024-09-13T05:19:26.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443856+0000", 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7648000000000001e-05 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.7785635690000001, 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:19:26.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:26.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:19:26.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:19:26.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:19:26.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4797.0:1 3.3 e48/47 ondisk, result = 0)", 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.461454+0000", 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.7698565349999997, 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00035433200000000001, 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461454+0000", 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461454+0000", 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461453+0000", 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461460+0000", 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3829999999999997e-06 2024-09-13T05:19:26.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461476+0000", 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6229999999999999e-05 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461483+0000", 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3919999999999997e-06 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461687+0000", 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020401100000000001 2024-09-13T05:19:26.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461696+0000", 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1480000000000006e-06 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461809+0000", 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112971 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.334 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:26.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:26.337 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:26.330+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:rbd-nbd: the device /dev/nbd0 is busy 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ dev1=/dev/nbd0 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 21740 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=21740 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:26.338 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:26.339 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:26.375 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:26.375 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^21740\b' 2024-09-13T05:19:26.375 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^21740\b' 2024-09-13T05:19:26.424 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:26.424 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:26.424 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *21740\b' 2024-09-13T05:19:26.424 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *21740\b' 2024-09-13T05:19:26.430 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:26.430 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:26.436 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:26.438+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:26.512 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:26.513 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:26.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:26.574 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:26.570+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:26.638 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:26.634+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:26.652 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:26.652 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:26.652 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:26.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:26.758 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:26.754+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:26.841 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:26.838+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.887674132000001, 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.842 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:19:26.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.887466974000006, 2024-09-13T05:19:26.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:19:26.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:19:26.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.708957733999995, 2024-09-13T05:19:26.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:19:26.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:19:26.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.860 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.324577411000007, 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:19:26.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:19:26.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:19:26.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:19:26.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:19:26.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.324540275000004, 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.867 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:19:26.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:26.873 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.324434355999998, 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.324162330999997, 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:26.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:26.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:26.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:26.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.324118745000007, 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:26.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.324086518000001, 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.891 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:26.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:26.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.323875369999996, 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:26.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:26.902 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.323839077000002, 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.903 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.908 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.323800626999997, 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:26.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:26.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:26.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.323625724999999, 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:26.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:26.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:26.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:26.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.5089217870000002, 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:19:26.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.5013916739999997, 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.930 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:19:26.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4918223470000003, 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.935 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:19:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:19:26.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4806594750000004, 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:26.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:19:26.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:19:26.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4744998779999996, 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.946 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:19:26.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:19:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4256640980000004, 2024-09-13T05:19:26.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:19:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4173875640000002, 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:26.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:26.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:19:26.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.963 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:26.963 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:26.963 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:26.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:26.965 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:26.965 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21435 2024-09-13T05:19:26.966 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:26.966 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:26.966 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:26.966 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd-nbd 2024-09-13T05:19:26.966 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd-nbd 2024-09-13T05:19:26.966 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:26.966 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21435 2024-09-13T05:19:26.966 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:26.966+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:27.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:27.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' /dev/nbd0 = /dev/nbd0 ']' 2024-09-13T05:19:27.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:27.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep testrbdnbd21435 2024-09-13T05:19:27.065 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:27.062+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:27.103 INFO:tasks.workunit.client.0.smithi122.stdout:21836 rbd testrbdnbd21435 - /dev/nbd0 2024-09-13T05:19:27.103 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:27.103 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:27.103 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:27.104 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:27.104 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:27.142 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:27.142 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:27.142 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:27.142 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:27.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:27.166 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:27.166+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:27.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=21836 2024-09-13T05:19:27.171 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 21836 2024-09-13T05:19:27.172 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 21836 -C rbd-nbd 2024-09-13T05:19:27.177 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:27.177 INFO:tasks.workunit.client.0.smithi122.stdout: 21836 ? 00:00:00 rbd-nbd 2024-09-13T05:19:27.178 INFO:tasks.workunit.client.0.smithi122.stderr:++ dd if=/tmp/tmp.WvgaeitFGU/data bs=1M 2024-09-13T05:19:27.178 INFO:tasks.workunit.client.0.smithi122.stderr:++ md5sum 2024-09-13T05:19:27.268 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:27.266+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:27.293 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-09-13T05:19:27.293 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-09-13T05:19:27.293 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.113957 s, 589 MB/s 2024-09-13T05:19:27.297 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo dd if=/dev/nbd0 bs=1M 2024-09-13T05:19:27.297 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:27.298 INFO:tasks.workunit.client.0.smithi122.stderr:++ md5sum 2024-09-13T05:19:27.298 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:27.298 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:27.298 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which dd 2024-09-13T05:19:27.298 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/dd 2024-09-13T05:19:27.298 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:27.298 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/dd if=/dev/nbd0 bs=1M 2024-09-13T05:19:27.325 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:27.326 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:27.326 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:27.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:27.369 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:27.370+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:27.470 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:27.466+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:27.486 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:27.486 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:27.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:27.486 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:27.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.486 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4190.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-13T05:19:27.486 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:27.486 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.350683712999995, 2024-09-13T05:19:27.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000925975, 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132151+0000", 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132150+0000", 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132161+0000", 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0951999999999999e-05 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132177+0000", 2024-09-13T05:19:27.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5316999999999999e-05 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132188+0000", 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1238e-05 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132237+0000", 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8776000000000002e-05 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.132282+0000", 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5683999999999998e-05 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133035+0000", 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075311499999999995 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:16.133077+0000", 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1356000000000002e-05 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.1551785790000002, 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.491 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:19:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.1439728179999999, 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:27.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:19:27.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.133793764, 2024-09-13T05:19:27.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:19:27.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:27.505 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.1263579610000001, 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:27.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:19:27.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:27.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:27.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.511 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.1197502579999998, 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.512 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:19:27.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:19:27.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:19:27.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:19:27.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:19:27.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:19:27.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:19:27.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:19:27.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:19:27.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:27.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:19:27.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:19:27.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:19:27.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:19:27.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:19:27.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:19:27.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.519 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.1105240399999996, 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:19:27.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4797.0:1 3.3 e48/47 3:c4c92e5a:::rbd_trash:head v 48'1, mlcod=0'0)", 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.0230301260000001, 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015947019999999999, 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459810+0000", 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8649999999999998e-06 2024-09-13T05:19:27.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459820+0000", 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9899999999999992e-06 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459829+0000", 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.431e-06 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459885+0000", 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5868000000000001e-05 2024-09-13T05:19:27.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459912+0000", 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7203e-05 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461364+0000", 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014515820000000001 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461399+0000", 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5063e-05 2024-09-13T05:19:27.528 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4664.0:12 2.3 e50/16 2:d073633c:::rbd_header.11d2ef348a4:head v 50'34, mlcod=50'32)", 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:26.083351+0000", 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.3994833959999999, 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000923675, 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.083351+0000", 2024-09-13T05:19:27.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.083351+0000", 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.083349+0000", 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.083373+0000", 2024-09-13T05:19:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4182e-05 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.083412+0000", 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9159000000000002e-05 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.083435+0000", 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2595999999999998e-05 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.083497+0000", 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2896000000000002e-05 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.083523+0000", 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5140999999999999e-05 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.084226+0000", 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070295399999999999 2024-09-13T05:19:27.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.084275+0000", 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9200000000000003e-05 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4664.0:119 2.3 e50/16 2:d073633c:::rbd_header.11d2ef348a4:head v 50'39, mlcod=50'37)", 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:26.364298+0000", 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.118536813, 2024-09-13T05:19:27.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013753039999999999, 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364298+0000", 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364298+0000", 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364296+0000", 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364309+0000", 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3246e-05 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364340+0000", 2024-09-13T05:19:27.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0216e-05 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364352+0000", 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2327000000000001e-05 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364454+0000", 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010179399999999999 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364497+0000", 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2916e-05 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.365601+0000", 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001104397 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.365673+0000", 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2047999999999999e-05 2024-09-13T05:19:27.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4679.0:12 2.3 e50/16 2:d073633c:::rbd_header.11d2ef348a4:head v 50'40, mlcod=50'38)", 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.016450+0000", 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.466384621, 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001286354, 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016450+0000", 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016450+0000", 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016447+0000", 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016472+0000", 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5015999999999999e-05 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016522+0000", 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9549000000000003e-05 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016545+0000", 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3275999999999999e-05 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016658+0000", 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112667 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016698+0000", 2024-09-13T05:19:27.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0626e-05 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.017663+0000", 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00096442499999999998 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.017736+0000", 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3480999999999997e-05 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 50/16 rep_tid 168 entries 50'41 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.4679.0:74 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 50'39)", 2024-09-13T05:19:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.314757+0000", 2024-09-13T05:19:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.16807712799999999, 2024-09-13T05:19:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010613650000000001, 2024-09-13T05:19:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.314757+0000", 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.314757+0000", 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.314755+0000", 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:27.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.314784+0000", 2024-09-13T05:19:27.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:19:27.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.314839+0000", 2024-09-13T05:19:27.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4789e-05 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.314874+0000", 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5542e-05 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.314956+0000", 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2059999999999997e-05 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.315819+0000", 2024-09-13T05:19:27.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00086253399999999998 2024-09-13T05:19:27.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.547 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.547 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 50/16 rep_tid 169 entries 50'42 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.4679.0:88 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 50'40)", 2024-09-13T05:19:27.547 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.328127+0000", 2024-09-13T05:19:27.547 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.15470779200000001, 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010504900000000001, 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.328127+0000", 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.328127+0000", 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.328124+0000", 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.328152+0000", 2024-09-13T05:19:27.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7251999999999999e-05 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.328208+0000", 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6271999999999999e-05 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.328245+0000", 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7373000000000003e-05 2024-09-13T05:19:27.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.328327+0000", 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.174e-05 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.329177+0000", 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00085014200000000002 2024-09-13T05:19:27.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.552 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.552 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:162 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:27.552 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.401274+0000", 2024-09-13T05:19:27.552 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.081560243000000004, 2024-09-13T05:19:27.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001023403, 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 162 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.401274+0000", 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.401274+0000", 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.401274+0000", 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.401290+0000", 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6824999999999999e-05 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.401303+0000", 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2262999999999999e-05 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.401311+0000", 2024-09-13T05:19:27.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5939999999999994e-06 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.401592+0000", 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000280845 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.401733+0000", 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014107900000000001 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.402298+0000", 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000564458 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:164 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.405680+0000", 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.077154080999999999, 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00105463, 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 164 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.405680+0000", 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.405680+0000", 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.405680+0000", 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.405685+0000", 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0300000000000001e-06 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.405696+0000", 2024-09-13T05:19:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1313e-05 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.405704+0000", 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9470000000000003e-06 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.406086+0000", 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038164399999999999 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.406218+0000", 2024-09-13T05:19:27.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132767 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.406735+0000", 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00051668799999999996 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:172 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.414655+0000", 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.068179983999999999, 2024-09-13T05:19:27.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001016021, 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 172 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.414655+0000", 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.414655+0000", 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.414654+0000", 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.414657+0000", 2024-09-13T05:19:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4079999999999998e-06 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.414668+0000", 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.076e-05 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.414691+0000", 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3017e-05 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.415287+0000", 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059558799999999998 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.415341+0000", 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4292999999999997e-05 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.415671+0000", 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032947199999999998 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:174 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 1576960~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.416253+0000", 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.066581346, 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00082069100000000002, 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.416253+0000", 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.416253+0000", 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.416252+0000", 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.416266+0000", 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4039999999999999e-05 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.416278+0000", 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1177e-05 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.416286+0000", 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:19:27.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.416637+0000", 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00035055299999999998 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.416713+0000", 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6656999999999995e-05 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.417074+0000", 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00036060500000000001 2024-09-13T05:19:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:180 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.058483994999999997, 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013961119999999999, 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:27.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 180 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424350+0000", 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424354+0000", 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3560000000000003e-06 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424365+0000", 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0832999999999999e-05 2024-09-13T05:19:27.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424374+0000", 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5599999999999994e-06 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424997+0000", 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00062386799999999997 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.425131+0000", 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133614 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.425747+0000", 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00061550899999999998 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:188 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.435086+0000", 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.047748507000000003, 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011240899999999999, 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 188 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.435086+0000", 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.435086+0000", 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.435085+0000", 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.435093+0000", 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5039999999999996e-06 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.435106+0000", 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3377000000000001e-05 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.435115+0000", 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2040000000000003e-06 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.435565+0000", 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044945299999999999 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.435699+0000", 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013459100000000001 2024-09-13T05:19:27.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.436210+0000", 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00051088800000000003 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:193 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 4067328~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.438665+0000", 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.044169841000000001, 2024-09-13T05:19:27.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00087516899999999997, 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 193 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.438665+0000", 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.438665+0000", 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.438664+0000", 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.438669+0000", 2024-09-13T05:19:27.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2000000000000002e-06 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.438681+0000", 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2123999999999999e-05 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.438689+0000", 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2460000000000003e-06 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:27.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.438938+0000", 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024804699999999999 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.439067+0000", 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129794 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.439540+0000", 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047249999999999999 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.583 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.584 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.584 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:27.584 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:27.584 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:27.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:19:27.586 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:27.587 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:27.582+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:27.686 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:27.682+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:27.716 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:27.716 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:27.730 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:27.787 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:27.782+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:27.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:27.856 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:27.856 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:27.856 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:27.856 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:27.857 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:27.871 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:27.902 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:27.898+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:27.984 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-09-13T05:19:27.984 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-09-13T05:19:27.984 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.680238 s, 98.7 MB/s 2024-09-13T05:19:27.987 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 'df5d783dd8bb3acc7f54c36327a417dd -' = 'df5d783dd8bb3acc7f54c36327a417dd -' ']' 2024-09-13T05:19:27.987 INFO:tasks.workunit.client.0.smithi122.stderr:+ dd if=/dev/urandom of=/tmp/tmp.WvgaeitFGU/data bs=1M count=64 2024-09-13T05:19:27.989 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:27.986+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:27.992 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:27.992 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:27.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:27.992 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:27.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.992 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:27.992 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:27.992 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.859844475000003, 2024-09-13T05:19:27.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:27.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:27.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:27.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.859213216000001, 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:27.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.858568094999995, 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.001 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:28.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:28.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.858060731999998, 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.010 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.857535295999995, 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:28.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.856897783999997, 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:28.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.019 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.856321076, 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:28.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:28.024 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.024 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.024 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.024 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:28.024 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:28.024 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.855769602999999, 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:28.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:28.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:28.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:28.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.855309019999993, 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.030 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:28.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:28.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.854799134000004, 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:28.034 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:28.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:28.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.038 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.854372071, 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:28.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:28.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.467612353000007, 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:28.043 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:28.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 76.850077541000005, 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.048 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.052 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 6.6192986060000001, 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:19:28.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:19:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 6.6086782619999997, 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:28.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:19:28.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 6.6014639480000001, 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:19:28.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:19:28.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:19:28.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 6.5951084279999996, 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:28.068 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:19:28.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:19:28.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:19:28.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:28.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:19:28.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:19:28.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:28.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:19:28.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:19:28.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:28.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:19:28.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:19:28.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:19:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 6.5950520639999999, 2024-09-13T05:19:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:19:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:19:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:19:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:19:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:19:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:19:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:19:28.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:19:28.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:19:28.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:19:28.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:28.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:19:28.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:19:28.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:19:28.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:19:28.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:19:28.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:19:28.083 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.083 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.084 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.084 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:28.084 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:28.084 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 6.5950144929999999, 2024-09-13T05:19:28.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:19:28.084 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:19:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 6.5949799689999997, 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:19:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:19:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:28.095 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:28.096 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:19:28.097 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:28.098 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:28.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:28.099 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:28.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:28.100 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:28.100 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:28.100 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:28.101 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:28.101 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:28.101 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:28.090+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:28.198 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:28.194+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:28.269 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:28.269 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:28.283 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:28.298 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:28.294+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:28.399 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:28.394+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:28.404 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:28.405 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:28.418 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:28.486 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 5, 3] 2024-09-13T05:19:28.486 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-09-13T05:19:28.486 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T05:19:28.487 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T05:19:28.489 INFO:tasks.ceph.osd.4:Started 2024-09-13T05:19:28.489 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:28.535 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:28.533+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:28.537 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:28.537 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:28.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:28.583 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-09-13T05:19:28.583 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-09-13T05:19:28.583 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.582066 s, 115 MB/s 2024-09-13T05:19:28.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-13T05:19:28.584 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:28.584 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:28.586 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:28.587 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:19:28.589 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:19:28.589 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:28.589 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-13T05:19:28.602 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:28.598+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-09-13T05:19:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:0 2024-09-13T05:19:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:1 2024-09-13T05:19:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:2 2024-09-13T05:19:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:abort 2024-09-13T05:19:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:assert 2024-09-13T05:19:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:config diff 2024-09-13T05:19:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:config diff get 2024-09-13T05:19:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-09-13T05:19:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:config help [] 2024-09-13T05:19:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-09-13T05:19:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:admin_socket: invalid command 2024-09-13T05:19:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T05:19:28.613 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-13T05:19:28.702 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:28.698+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:28.706 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:28.707 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:28.723 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:28.814 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:28.810+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:28.904 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:28.898+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:28.910 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:28.910 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:28.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:29.013 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:29.010+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:29.102 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:29.102 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:29.105 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:29.101+0000 7f5357da97c0 -1 Falling back to public interface 2024-09-13T05:19:29.106 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:29.102+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:29.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:29.206 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:29.205+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:29.221 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-09-13T05:19:29.221 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-09-13T05:19:29.221 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.622585 s, 108 MB/s 2024-09-13T05:19:29.224 INFO:tasks.workunit.client.0.smithi122.stderr:++ dd if=/tmp/tmp.WvgaeitFGU/data bs=1M 2024-09-13T05:19:29.224 INFO:tasks.workunit.client.0.smithi122.stderr:++ md5sum 2024-09-13T05:19:29.303 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:29.304 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:29.307 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:29.302+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:29.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:29.359 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-09-13T05:19:29.359 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-09-13T05:19:29.359 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.133079 s, 504 MB/s 2024-09-13T05:19:29.360 INFO:tasks.workunit.client.0.smithi122.stderr:++ md5sum 2024-09-13T05:19:29.360 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --no-progress export testrbdnbd21435 - 2024-09-13T05:19:29.428 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:29.422+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:29.444 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:19:29.439+0000 7fdd2aea2640 0 -- 172.21.15.122:0/1329249224 >> [v2:172.21.15.16:6800/2153409514,v1:172.21.15.16:6801/2153409514] conn(0x7fdd04004b90 msgr2=0x7fdd04006f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T05:19:29.448 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:19:29.443+0000 7fdd2c104640 0 -- 172.21.15.122:0/1329249224 >> [v2:172.21.15.16:6808/2608485151,v1:172.21.15.16:6809/2608485151] conn(0x55ba8a2432d0 msgr2=0x55ba8a2456c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T05:19:29.506 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:29.506 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:29.509 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:29.509+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:29.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:29.610 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:29.606+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:29.622 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' '15c53db987cbfaea4af4b6d33f90ca20 -' = '15c53db987cbfaea4af4b6d33f90ca20 -' ']' 2024-09-13T05:19:29.622 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 21836 2024-09-13T05:19:29.622 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:19:29.622 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=21836 2024-09-13T05:19:29.622 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:29.623 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:29.623 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:29.623 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:29.624 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:29.625 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:29.625 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:29.625 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:29.692 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:29.692 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:29.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:29.717 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:29.717 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^21836\b' 2024-09-13T05:19:29.717 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^21836\b' 2024-09-13T05:19:29.754 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:29.750+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:29.765 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:29.766 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:29.766 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *21836\b' 2024-09-13T05:19:29.766 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *21836\b' 2024-09-13T05:19:29.772 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:29.772 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:29.811 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:29.806+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:29.882 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:29.882 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:29.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:29.938 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:29.934+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:30.012 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:30.006+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:30.062 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:30.063 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:30.077 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:30.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:19:30.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:30.113 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:30.109+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:30.214 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:30.210+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:30.240 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:30.240 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:30.260 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:30.274 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --options notrim map rbd/testrbdnbd21435 2024-09-13T05:19:30.275 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:30.275 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:30.276 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:30.276 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:30.277 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:30.278 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:30.279 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:30.279 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options notrim map rbd/testrbdnbd21435 2024-09-13T05:19:30.293 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:30.309 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:30.314 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:30.310+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:30.328 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:30.329 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:30.348 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:30.349 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:30.352 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:30.390 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:30.390 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:30.390 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:30.390 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:30.404 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:30.415 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:30.410+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:30.418 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:30.418 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:30.419 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:30.419 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:30.419 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:30.419 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:30.487 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=21946 2024-09-13T05:19:30.487 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 21946 2024-09-13T05:19:30.487 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 21946 -C rbd-nbd 2024-09-13T05:19:30.493 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:30.493 INFO:tasks.workunit.client.0.smithi122.stdout: 21946 ? 00:00:00 rbd-nbd 2024-09-13T05:19:30.494 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd21435 2024-09-13T05:19:30.494 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-13T05:19:30.516 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:30.514+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:30.524 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:30.524 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:30.525 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:30.525 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:30.525 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:30.539 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:30.553 INFO:tasks.workunit.client.0.smithi122.stderr:+ provisioned=67108864 2024-09-13T05:19:30.554 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd21435 2024-09-13T05:19:30.554 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-13T05:19:30.612 INFO:tasks.workunit.client.0.smithi122.stderr:+ used=67108864 2024-09-13T05:19:30.612 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-13T05:19:30.612 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo blkdiscard /dev/nbd0 2024-09-13T05:19:30.612 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-13T05:19:30.612 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:30.612 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:30.613 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:30.613 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-09-13T05:19:30.614 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-13T05:19:30.614 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:30.614 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-13T05:19:30.617 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:30.613+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:30.628 INFO:tasks.workunit.client.0.smithi122.stderr:blkdiscard: /dev/nbd0: BLKDISCARD ioctl failed: Operation not supported 2024-09-13T05:19:30.628 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:30.629 INFO:tasks.workunit.client.0.smithi122.stderr:+ sync 2024-09-13T05:19:30.664 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.530437528999997, 2024-09-13T05:19:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.529806269999995, 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.529161149000004, 2024-09-13T05:19:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.528653786000007, 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.528128350000003, 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.527490838000006, 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.526914129999994, 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.526362656999993, 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.525902074000001, 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.525392187999998, 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.524965124999994, 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 80.138205407000001, 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.520670594999999, 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.2898916600000003, 2024-09-13T05:19:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:19:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:19:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.2792713159999991, 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:19:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.2720570020000004, 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:19:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:19:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.2657014820000008, 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:19:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:19:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:19:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.2656451180000001, 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:19:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.2656075470000001, 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:19:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:19:30.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:19:30.762 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.265573023, 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.764 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:19:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:19:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:30.769 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:30.771 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:30.772 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:30.766+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:30.871 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:30.869+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:30.940 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:30.941 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:30.941 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:30.941 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:30.954 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:30.972 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:30.966+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:19:31.106 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:31.102+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:31.111 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:31.111 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:31.111 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:31.112 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:31.112 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:31.126 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:31.174 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:31.174+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:31.248 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.9061781979999992, 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:19:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:19:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:19:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8942975959999995, 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:19:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.888353897, 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:19:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8856781890000001, 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:19:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:19:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:19:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8809641559999992, 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:49 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'17, mlcod=46'15)", 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.379251+0000", 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8677817789999995, 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001378642, 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379251+0000", 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379251+0000", 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379249+0000", 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379264+0000", 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5752000000000001e-05 2024-09-13T05:19:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379291+0000", 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6375000000000001e-05 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379306+0000", 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.543e-05 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379411+0000", 2024-09-13T05:19:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000104886 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.379448+0000", 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6897999999999998e-05 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.380537+0000", 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001088901 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.380629+0000", 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2287000000000006e-05 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8650118829999993, 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:19:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:19:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8553716809999994, 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:19:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:19:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8482788760000002, 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:19:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8400609199999991, 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:19:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:19:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8379151070000006, 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:19:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:19:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.835997313, 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:56 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'24, mlcod=46'17)", 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.412806+0000", 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.8342265690000001, 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001612657, 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412806+0000", 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412806+0000", 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412804+0000", 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412819+0000", 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4842e-05 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412847+0000", 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7458999999999999e-05 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412864+0000", 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7141000000000001e-05 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413020+0000", 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000156346 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413060+0000", 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9774999999999998e-05 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414357+0000", 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012974569999999999 2024-09-13T05:19:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414418+0000", 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1062999999999995e-05 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 3.6825510509999999, 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:19:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:19:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:19:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 3.6761131110000003, 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:19:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:19:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 3.6700449580000001, 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:19:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:302 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1052672~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.578551+0000", 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 3.668481382, 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001380165, 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 302 2024-09-13T05:19:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578551+0000", 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578551+0000", 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578549+0000", 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578560+0000", 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1453e-05 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578587+0000", 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6628999999999999e-05 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578601+0000", 2024-09-13T05:19:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3686000000000001e-05 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578759+0000", 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000158395 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578962+0000", 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020303300000000001 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.579931+0000", 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096897300000000003 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 3.662501405, 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:19:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:306 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1576960~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.587358+0000", 2024-09-13T05:19:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 3.6596741779999999, 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001418494, 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 306 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.587358+0000", 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.587358+0000", 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.587356+0000", 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.587369+0000", 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2719e-05 2024-09-13T05:19:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.587401+0000", 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1878999999999998e-05 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.587415+0000", 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4039000000000001e-05 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.587995+0000", 2024-09-13T05:19:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00057965299999999996 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.588150+0000", 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00015536200000000001 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.588777+0000", 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00062677800000000004 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 3.6498241600000001, 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:31.335 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:19:31.338 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.339 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.339 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.339 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:31.339 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:31.339 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:31.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:31.341 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:31.334+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:31.442 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:31.438+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:31.475 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:31.475 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:31.475 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:31.475 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:31.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:31.578 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:31.574+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:31.643 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:31.638+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:31.662 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:31.662 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:31.662 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:31.662 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:31.662 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:31.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:31.762 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:31.758+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:31.846 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:31.842+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:31.847 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:31.847 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:31.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:31.847 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:31.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.710656290000003, 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:31.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:19:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:19:31.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.700874760000005, 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:19:31.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:19:31.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:19:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.478044507, 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:19:31.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:19:31.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:31.865 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.471014911999999, 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:19:31.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:31.870 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.430232076999999, 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.875 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.423373604, 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:19:31.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:64 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [watch unwatch cookie 139805748911888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.401536468, 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012844899999999999, 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 64 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442441+0000", 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442448+0000", 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6540000000000002e-06 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442465+0000", 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7164999999999999e-05 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442472+0000", 2024-09-13T05:19:31.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2599999999999999e-06 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442630+0000", 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015837800000000001 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442736+0000", 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000105449 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443603+0000", 2024-09-13T05:19:31.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00086757000000000002 2024-09-13T05:19:31.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:31.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443639+0000", 2024-09-13T05:19:31.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5806999999999998e-05 2024-09-13T05:19:31.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443727+0000", 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7646999999999996e-05 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.3912316429999994, 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:19:31.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:19:31.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:19:31.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:19:31.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:31.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:98 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.5034736859999995, 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001341128, 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 98 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:31.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340508+0000", 2024-09-13T05:19:31.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1829999999999998e-06 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340515+0000", 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4279999999999999e-06 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340520+0000", 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9309999999999999e-06 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341329+0000", 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080829399999999998 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341460+0000", 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131271 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341846+0000", 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038646500000000001 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.897 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:101 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [sparse-read 659456~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.342590+0000", 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.5013888389999996, 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012068000000000001, 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 101 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.898 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342590+0000", 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342590+0000", 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342589+0000", 2024-09-13T05:19:31.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342596+0000", 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1640000000000004e-06 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342616+0000", 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9879000000000001e-05 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342631+0000", 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4922999999999999e-05 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342996+0000", 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000365125 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.343185+0000", 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000189495 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.343797+0000", 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000611324 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:230 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.3574087400000003, 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001614084, 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 230 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486575+0000", 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3340000000000001e-06 2024-09-13T05:19:31.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486581+0000", 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4689999999999996e-06 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486584+0000", 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8010000000000001e-06 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.487120+0000", 2024-09-13T05:19:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053554000000000004 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.487394+0000", 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027411100000000001 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.488184+0000", 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079022900000000004 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:233 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 659456~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.3547476740000004, 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014141659999999999, 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 233 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489234+0000", 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2660000000000002e-06 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489241+0000", 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6070000000000002e-06 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489246+0000", 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.656e-06 2024-09-13T05:19:31.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489607+0000", 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000360711 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489893+0000", 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028591100000000002 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.490645+0000", 2024-09-13T05:19:31.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075255999999999999 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:236 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1052672~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.3522685729999999, 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013904989999999999, 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.911 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491709+0000", 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491716+0000", 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491733+0000", 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7652999999999999e-05 2024-09-13T05:19:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491742+0000", 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1800000000000002e-06 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.492084+0000", 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341479 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.492348+0000", 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026391400000000002 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.493101+0000", 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075303000000000002 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:238 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.3466481049999999, 2024-09-13T05:19:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015224399999999999, 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 238 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497330+0000", 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497333+0000", 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2030000000000001e-06 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497341+0000", 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4200000000000001e-06 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497345+0000", 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1679999999999997e-06 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497879+0000", 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053369599999999995 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.498107+0000", 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000228435 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.498853+0000", 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074610300000000002 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:241 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1708032~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.500034+0000", 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.3439443649999996, 2024-09-13T05:19:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011367669999999999, 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 241 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500034+0000", 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500034+0000", 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500032+0000", 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500043+0000", 2024-09-13T05:19:31.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0716e-05 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500062+0000", 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9403e-05 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500072+0000", 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9569999999999997e-06 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500178+0000", 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010554 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500436+0000", 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025817800000000002 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.501171+0000", 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00073542300000000002 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.3360763560000004, 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012715370000000001, 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:31.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507905+0000", 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9409999999999999e-06 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507914+0000", 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8729999999999996e-06 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507918+0000", 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9009999999999996e-06 2024-09-13T05:19:31.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.508538+0000", 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00062024300000000003 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.508662+0000", 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012431000000000001 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.509174+0000", 2024-09-13T05:19:31.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00051175600000000004 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:254 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.517640+0000", 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.326338969, 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001145924, 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 254 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517640+0000", 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517640+0000", 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517639+0000", 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517644+0000", 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517657+0000", 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2846e-05 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517667+0000", 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5729999999999997e-06 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.518126+0000", 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045909999999999999 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.518289+0000", 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016252899999999999 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.518786+0000", 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049729899999999998 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:261 2.2 2:59ccbc16:::rbd_data.11d2ef348a4.0000000000000006:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.3223340759999997, 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001208492, 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.932 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521644+0000", 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521646+0000", 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8950000000000001e-06 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521671+0000", 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4397999999999999e-05 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521675+0000", 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4329999999999997e-06 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522277+0000", 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060205000000000002 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522382+0000", 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104731 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522853+0000", 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047135200000000002 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.936 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:287 2.2 2:59ccbc16:::rbd_data.11d2ef348a4.0000000000000006:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.2871186999999997, 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014822030000000001, 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 287 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556859+0000", 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556864+0000", 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556878+0000", 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3457e-05 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556885+0000", 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9060000000000002e-06 2024-09-13T05:19:31.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.557685+0000", 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079915800000000001 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.557810+0000", 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000125084 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.558342+0000", 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053261499999999995 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:411 2.6 2:695f0dc1:::rbd_data.11d2ef348a4.000000000000000a:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.734652+0000", 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1093268119999999, 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001201894, 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:31.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 411 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734652+0000", 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734652+0000", 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:31.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734651+0000", 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734657+0000", 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734668+0000", 2024-09-13T05:19:31.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1559999999999999e-05 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734676+0000", 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3599999999999998e-06 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.735325+0000", 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00064975799999999995 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:31.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.735455+0000", 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012965700000000001 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.735854+0000", 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039879599999999998 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:31.945 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:31.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:31.947 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:31.945+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:32.048 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:32.045+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:32.121 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:32.121 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:32.121 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:32.121 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:32.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:32.149 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:32.145+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:32.250 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:32.249+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:32.264 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:32.264 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:32.264 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:32.264 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:32.264 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:32.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:32.351 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:32.349+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:32.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.447274909000001, 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:19:32.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:19:32.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:19:32.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.447067751000006, 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:19:32.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:19:32.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:19:32.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268558510999995, 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:19:32.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:19:32.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:19:32.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:19:32.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:19:32.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:19:32.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:19:32.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.884178188000007, 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:32.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:19:32.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.427 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.884141052000004, 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:19:32.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:19:32.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:19:32.433 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.884035132999998, 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.434 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:32.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:32.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:32.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.442 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.883763107999997, 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:32.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:32.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:32.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:32.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.883719522000007, 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:32.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.883687295000001, 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:32.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.883476146999996, 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:32.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.465 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.883439854000002, 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:32.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.883401403999997, 2024-09-13T05:19:32.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:32.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:32.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 81.883226501999999, 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.477 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:32.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:32.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:32.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:32.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:32.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:32.483 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.068522564, 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:19:32.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:19:32.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.060992451000001, 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:19:32.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:19:32.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.051423123999999, 2024-09-13T05:19:32.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:19:32.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.499 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.040260251999999, 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:19:32.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:19:32.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.034100655, 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:19:32.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:19:32.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.985264875, 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:19:32.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:19:32.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.976988341, 2024-09-13T05:19:32.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:19:32.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:19:32.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:32.521 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:32.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:32.523 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:32.524 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:32.525 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:32.518+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:32.529 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:32.529 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:32.550 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:32.550 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:32.550 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T05:19:32.568 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd21435 2024-09-13T05:19:32.568 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-13T05:19:32.571 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:32.571 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:32.576 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T05:19:32.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T05:19:32.623 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:32.618+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:32.628 INFO:tasks.workunit.client.0.smithi122.stderr:+ provisioned=67108864 2024-09-13T05:19:32.629 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd21435 2024-09-13T05:19:32.629 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-13T05:19:32.647 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:32.647 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:32.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:32.688 INFO:tasks.workunit.client.0.smithi122.stderr:+ used=67108864 2024-09-13T05:19:32.688 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-13T05:19:32.688 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 21946 2024-09-13T05:19:32.688 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:19:32.688 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=21946 2024-09-13T05:19:32.688 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:32.688 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:32.688 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:32.689 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:32.689 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:32.690 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:32.690 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:32.690 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:32.750 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:32.746+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:32.763 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:32.763 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^21946\b' 2024-09-13T05:19:32.763 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^21946\b' 2024-09-13T05:19:32.809 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:32.810 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:32.810 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *21946\b' 2024-09-13T05:19:32.810 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *21946\b' 2024-09-13T05:19:32.815 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:32.816 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:32.825 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:32.821+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:32.833 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:32.833 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:32.850 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:32.938 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:32.934+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:32.950 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:32.945+0000 7f5357da97c0 -1 osd.4 38 log_to_monitors true 2024-09-13T05:19:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:33.026 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:33.022+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:33.036 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:33.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:19:33.100 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:33.097+0000 7f534bb70640 -1 osd.4 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T05:19:33.128 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:33.125+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:33.168 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:33.168 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:33.182 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:33.229 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:33.222+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:33.318 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:33.318 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:33.318 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:33.319 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:33.320 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:33.322 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:33.322 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:33.322 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:33.330 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:33.325+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:33.343 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:33.344 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:33.358 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:33.431 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:33.426+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:33.463 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:33.463 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:33.463 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:33.463 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:33.464 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:33.464 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:33.519 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:33.519 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:33.531 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:33.530+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:33.532 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=22103 2024-09-13T05:19:33.532 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 22103 2024-09-13T05:19:33.532 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 22103 -C rbd-nbd 2024-09-13T05:19:33.533 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:33.538 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:33.538 INFO:tasks.workunit.client.0.smithi122.stdout: 22103 ? 00:00:00 rbd-nbd 2024-09-13T05:19:33.539 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd21435 2024-09-13T05:19:33.539 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-13T05:19:33.595 INFO:tasks.workunit.client.0.smithi122.stderr:+ provisioned=67108864 2024-09-13T05:19:33.596 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd21435 2024-09-13T05:19:33.596 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-13T05:19:33.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:33.651 INFO:tasks.workunit.client.0.smithi122.stderr:+ used=67108864 2024-09-13T05:19:33.651 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-13T05:19:33.651 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-13T05:19:33.651 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:33.651 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:33.652 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:33.652 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-09-13T05:19:33.653 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-13T05:19:33.653 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:33.653 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-13T05:19:33.656 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:33.656 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:33.657 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:33.653+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:33.671 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:33.736 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:33.736 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:33.736 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:33.736 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:33.737 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:33.733+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:33.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T05:19:33.752 INFO:tasks.workunit.client.0.smithi122.stderr:+ sync 2024-09-13T05:19:33.835 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:33.833+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:33.845 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:33.845 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:33.860 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:33.921 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd21435 2024-09-13T05:19:33.921 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-13T05:19:33.947 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:33.945+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:33.988 INFO:tasks.workunit.client.0.smithi122.stderr:+ provisioned=67108864 2024-09-13T05:19:33.989 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd -p rbd --format xml du testrbdnbd21435 2024-09-13T05:19:33.989 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-13T05:19:34.027 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:34.027 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:34.036 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:34.033+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:34.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:34.046 INFO:tasks.workunit.client.0.smithi122.stderr:+ used=0 2024-09-13T05:19:34.046 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 0 -lt 67108864 ']' 2024-09-13T05:19:34.046 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 22103 2024-09-13T05:19:34.046 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:19:34.046 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=22103 2024-09-13T05:19:34.046 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:34.047 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:34.047 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:34.047 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:34.047 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:34.048 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:34.048 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:34.048 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:34.056 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:19:34.126 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:34.126 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^22103\b' 2024-09-13T05:19:34.126 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^22103\b' 2024-09-13T05:19:34.137 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:34.134+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:34.174 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:34.174 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:34.175 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *22103\b' 2024-09-13T05:19:34.175 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *22103\b' 2024-09-13T05:19:34.181 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:34.181 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:34.206 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:34.207 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:34.207 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:34.207 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:34.221 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:34.267 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:34.265+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:34.337 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:34.337 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:34.337 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:34.337 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:34.338 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:34.339 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:34.334+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:34.351 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:34.439 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:34.437+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:34.520 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:34.521 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:34.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:34.521 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [] 2024-09-13T05:19:34.521 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:34.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:34.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:34.579 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:34.577+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:34.641 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:34.637+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:34.672 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:34.672 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:34.683 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device -t nbd -o try-netlink map rbd/testrbdnbd21435 2024-09-13T05:19:34.683 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:34.683 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:34.685 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:34.685 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:34.687 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:34.687 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:34.687 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device -t nbd -o try-netlink map rbd/testrbdnbd21435 2024-09-13T05:19:34.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:34.734 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:34.734 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:34.741 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:34.738+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:34.750 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:34.750 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:34.766 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:34.767 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:34.787 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:34.787 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:34.808 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T05:19:34.815 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:34.815 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:34.815 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:34.815 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:34.815 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:34.829 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:34.829 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:34.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:34.845 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:34.846 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:34.846 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:34.846 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:34.846 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:34.846 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:34.847 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:34.842+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:34.912 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=22259 2024-09-13T05:19:34.912 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 22259 2024-09-13T05:19:34.912 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 22259 -C rbd-nbd 2024-09-13T05:19:34.918 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:34.919 INFO:tasks.workunit.client.0.smithi122.stdout: 22259 ? 00:00:00 rbd-nbd 2024-09-13T05:19:34.919 INFO:tasks.workunit.client.0.smithi122.stderr:++ basename /dev/nbd0 2024-09-13T05:19:34.919 INFO:tasks.workunit.client.0.smithi122.stderr:+ devname=nbd0 2024-09-13T05:19:34.920 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-13T05:19:34.922 INFO:tasks.workunit.client.0.smithi122.stderr:+ blocks=65536 2024-09-13T05:19:34.922 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 65536 2024-09-13T05:19:34.922 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd resize rbd/testrbdnbd21435 --size 128M 2024-09-13T05:19:34.943 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:34.938+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:34.981 INFO:tasks.workunit.client.0.smithi122.stderr: Resizing image: 100% complete...done. 2024-09-13T05:19:34.993 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd info rbd/testrbdnbd21435 2024-09-13T05:19:35.022 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:35.022 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.887139938999994, 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:35.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:35.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:19:35.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:19:35.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:19:35.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:19:35.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:19:35.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:19:35.030 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 76.877358408999996, 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:19:35.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.654528156, 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.647498561000001, 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:19:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:19:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.606715726000001, 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:19:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:19:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.599857253, 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:19:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:19:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:19:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:64 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [watch unwatch cookie 139805748911888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.578020116999999, 2024-09-13T05:19:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012844899999999999, 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 64 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442441+0000", 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442448+0000", 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6540000000000002e-06 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442465+0000", 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7164999999999999e-05 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442472+0000", 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2599999999999999e-06 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442630+0000", 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015837800000000001 2024-09-13T05:19:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442736+0000", 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000105449 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443603+0000", 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00086757000000000002 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443639+0000", 2024-09-13T05:19:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5806999999999998e-05 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443727+0000", 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7646999999999996e-05 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.567715292000001, 2024-09-13T05:19:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:19:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:19:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:19:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:98 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.6799573350000001, 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001341128, 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 98 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340508+0000", 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1829999999999998e-06 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340515+0000", 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4279999999999999e-06 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340520+0000", 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9309999999999999e-06 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341329+0000", 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080829399999999998 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341460+0000", 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131271 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341846+0000", 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038646500000000001 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:101 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [sparse-read 659456~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.342590+0000", 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.6778724880000002, 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012068000000000001, 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 101 2024-09-13T05:19:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342590+0000", 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342590+0000", 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342589+0000", 2024-09-13T05:19:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342596+0000", 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1640000000000004e-06 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342616+0000", 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9879000000000001e-05 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342631+0000", 2024-09-13T05:19:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4922999999999999e-05 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342996+0000", 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000365125 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.343185+0000", 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000189495 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.343797+0000", 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000611324 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:230 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.533892389, 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001614084, 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 230 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486575+0000", 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3340000000000001e-06 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486581+0000", 2024-09-13T05:19:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4689999999999996e-06 2024-09-13T05:19:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486584+0000", 2024-09-13T05:19:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8010000000000001e-06 2024-09-13T05:19:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.487120+0000", 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053554000000000004 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.487394+0000", 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027411100000000001 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.488184+0000", 2024-09-13T05:19:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079022900000000004 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:233 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 659456~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.5312313230000001, 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014141659999999999, 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 233 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489234+0000", 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2660000000000002e-06 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489241+0000", 2024-09-13T05:19:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6070000000000002e-06 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489246+0000", 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.656e-06 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489607+0000", 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000360711 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489893+0000", 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028591100000000002 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.490645+0000", 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075255999999999999 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:236 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1052672~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.5287522219999996, 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013904989999999999, 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491709+0000", 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491716+0000", 2024-09-13T05:19:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491733+0000", 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7652999999999999e-05 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491742+0000", 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1800000000000002e-06 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.492084+0000", 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341479 2024-09-13T05:19:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.492348+0000", 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026391400000000002 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.493101+0000", 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075303000000000002 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:238 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.5231317539999996, 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015224399999999999, 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 238 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497330+0000", 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497333+0000", 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2030000000000001e-06 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497341+0000", 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4200000000000001e-06 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497345+0000", 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1679999999999997e-06 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497879+0000", 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053369599999999995 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.498107+0000", 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000228435 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.498853+0000", 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074610300000000002 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:241 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1708032~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.500034+0000", 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.5204280140000002, 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011367669999999999, 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 241 2024-09-13T05:19:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500034+0000", 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500034+0000", 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500032+0000", 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500043+0000", 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0716e-05 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500062+0000", 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9403e-05 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500072+0000", 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9569999999999997e-06 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500178+0000", 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010554 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.500436+0000", 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025817800000000002 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.501171+0000", 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00073542300000000002 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.5125600050000001, 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012715370000000001, 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507905+0000", 2024-09-13T05:19:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9409999999999999e-06 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507914+0000", 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8729999999999996e-06 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507918+0000", 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9009999999999996e-06 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.508538+0000", 2024-09-13T05:19:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00062024300000000003 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.508662+0000", 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012431000000000001 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.509174+0000", 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00051175600000000004 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:254 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.517640+0000", 2024-09-13T05:19:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.5028226179999997, 2024-09-13T05:19:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001145924, 2024-09-13T05:19:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.108 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 254 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517640+0000", 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517640+0000", 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517639+0000", 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517644+0000", 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517657+0000", 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2846e-05 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.517667+0000", 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5729999999999997e-06 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.518126+0000", 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045909999999999999 2024-09-13T05:19:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.518289+0000", 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016252899999999999 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.518786+0000", 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049729899999999998 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:261 2.2 2:59ccbc16:::rbd_data.11d2ef348a4.0000000000000006:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.4988177250000003, 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001208492, 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521644+0000", 2024-09-13T05:19:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521646+0000", 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8950000000000001e-06 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521671+0000", 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4397999999999999e-05 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521675+0000", 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4329999999999997e-06 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522277+0000", 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060205000000000002 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522382+0000", 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104731 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522853+0000", 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047135200000000002 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:287 2.2 2:59ccbc16:::rbd_data.11d2ef348a4.0000000000000006:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.4636023490000003, 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014822030000000001, 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 287 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556859+0000", 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556864+0000", 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556878+0000", 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3457e-05 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556885+0000", 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9060000000000002e-06 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.557685+0000", 2024-09-13T05:19:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079915800000000001 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.557810+0000", 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000125084 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.558342+0000", 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053261499999999995 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:411 2.6 2:695f0dc1:::rbd_data.11d2ef348a4.000000000000000a:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.734652+0000", 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.2858104609999996, 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001201894, 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 411 2024-09-13T05:19:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734652+0000", 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734652+0000", 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734651+0000", 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734657+0000", 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734668+0000", 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1559999999999999e-05 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.734676+0000", 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3599999999999998e-06 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.735325+0000", 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00064975799999999995 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.735455+0000", 2024-09-13T05:19:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012965700000000001 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.735854+0000", 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039879599999999998 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.126 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:35.127 INFO:tasks.workunit.client.0.smithi122.stdout:rbd image 'testrbdnbd21435': 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: size 128 MiB in 32 objects 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: order 22 (4 MiB objects) 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: snapshot_count: 0 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: id: 11d2ef348a4 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: block_name_prefix: rbd_data.11d2ef348a4 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: format: 2 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: op_features: 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: flags: 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: create_timestamp: Fri Sep 13 05:19:21 2024 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: access_timestamp: Fri Sep 13 05:19:21 2024 2024-09-13T05:19:35.128 INFO:tasks.workunit.client.0.smithi122.stdout: modify_timestamp: Fri Sep 13 05:19:21 2024 2024-09-13T05:19:35.130 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-13T05:19:35.130 INFO:tasks.workunit.client.0.smithi122.stderr:+ blocks2=131072 2024-09-13T05:19:35.130 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 131072 2024-09-13T05:19:35.130 INFO:tasks.workunit.client.0.smithi122.stderr:+ test 131072 -eq 131072 2024-09-13T05:19:35.130 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd resize rbd/testrbdnbd21435 --allow-shrink --size 64M 2024-09-13T05:19:35.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:35.131 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T05:19:35.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T05:19:35.132 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:35.122+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:35.139 INFO:tasks.workunit.client.0.smithi122.stderr: Resizing image: 50% complete... Resizing image: 53% complete... Resizing image: 56% complete... Resizing image: 59% complete... Resizing image: 62% complete... Resizing image: 65% complete... Resizing image: 68% complete... Resizing image: 71% complete... Resizing image: 75% complete... Resizing image: 78% complete... Resizing image: 81% complete... Resizing image: 84% complete... Resizing image: 87% complete... Resizing image: 90% complete... Resizing image: 93% complete... Resizing image: 96% complete... Resizing image: 100% complete...done. 2024-09-13T05:19:35.149 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-13T05:19:35.151 INFO:tasks.workunit.client.0.smithi122.stderr:+ blocks2=65536 2024-09-13T05:19:35.151 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 65536 2024-09-13T05:19:35.151 INFO:tasks.workunit.client.0.smithi122.stderr:+ test 65536 -eq 65536 2024-09-13T05:19:35.151 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 22259 2024-09-13T05:19:35.151 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:19:35.151 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=22259 2024-09-13T05:19:35.152 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:35.152 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:35.152 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:35.152 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:35.152 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:35.154 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:35.154 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:35.154 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:35.229 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:35.222+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:35.233 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:35.233 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^22259\b' 2024-09-13T05:19:35.233 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^22259\b' 2024-09-13T05:19:35.280 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:35.280 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:35.281 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *22259\b' 2024-09-13T05:19:35.281 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *22259\b' 2024-09-13T05:19:35.287 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:35.287 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:35.304 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:35.305 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:35.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:35.366 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:35.362+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:35.430 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:35.426+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:35.492 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:35.492 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:35.493 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:35.493 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:35.493 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:35.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:35.532 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:35.529+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:35.633 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:35.626+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:35.676 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:35.676 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:35.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:35.676 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:35.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.676 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.676 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.720417233999996, 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:19:35.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.682 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.720210076000001, 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:19:35.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:19:35.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.541700836000004, 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:19:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:19:35.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.157320513000002, 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:19:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:19:35.700 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.157283376999999, 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:19:35.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.706 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.157177458000007, 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:35.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.712 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.156905433000006, 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:35.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:35.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.156861847000002, 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:35.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:35.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:35.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.156829619999996, 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:35.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.729 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.156618472000005, 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:35.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:35.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:35.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:35.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.156582178999997, 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:35.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.741 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.156543729000006, 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:35.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:35.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:35.747 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.156368826999994, 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:35.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:35.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:35.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.341664889, 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.754 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:19:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:19:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:19:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.334134776000001, 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:19:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:19:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:19:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.324565449, 2024-09-13T05:19:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:19:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:19:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.313402577, 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:19:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.30724298, 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:19:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.258407200000001, 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.250130666, 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:19:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:35.791 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:35.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:35.794 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:35.790+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:35.794 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd --device-type nbd map --read-only rbd/testrbdnbd21435 2024-09-13T05:19:35.794 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:35.794 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:35.794 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:35.794 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:35.794 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:35.794 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:35.794 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --read-only rbd/testrbdnbd21435 2024-09-13T05:19:35.893 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:35.889+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:35.943 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:35.944 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd list 2024-09-13T05:19:35.944 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk -v pool=rbd -v img=testrbdnbd21435 -v dev=/dev/nbd0 '$2 == pool && $3 == img && $5 == dev {print $1}' 2024-09-13T05:19:35.975 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:35.975 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:35.975 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:35.975 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:35.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:35.994 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:35.989+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:36.007 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=22385 2024-09-13T05:19:36.007 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 22385 2024-09-13T05:19:36.007 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 22385 -C rbd-nbd 2024-09-13T05:19:36.014 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:36.014 INFO:tasks.workunit.client.0.smithi122.stdout: 22385 ? 00:00:00 rbd-nbd 2024-09-13T05:19:36.015 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-13T05:19:36.015 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:36.015 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:36.016 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:36.016 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:19:36.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:19:36.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:36.018 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-13T05:19:36.068 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T05:19:36.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:19:36.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T05:19:36.123 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:36.123 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:36.123 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:36.123 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:36.123 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:36.124 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:36.122+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:36.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:36.195 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:36.193+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:36.206 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-09-13T05:19:36.206 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-09-13T05:19:36.206 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.182817 s, 367 MB/s 2024-09-13T05:19:36.207 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-13T05:19:36.207 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-13T05:19:36.207 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:36.208 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:36.209 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:36.209 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:19:36.212 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:19:36.212 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:36.212 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-13T05:19:36.220 INFO:tasks.workunit.client.0.smithi122.stderr:/usr/bin/dd: error writing '/dev/nbd0': Operation not permitted 2024-09-13T05:19:36.220 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-09-13T05:19:36.220 INFO:tasks.workunit.client.0.smithi122.stderr:0+0 records out 2024-09-13T05:19:36.220 INFO:tasks.workunit.client.0.smithi122.stderr:0 bytes copied, 0.000765608 s, 0.0 kB/s 2024-09-13T05:19:36.221 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:36.222 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 22385 2024-09-13T05:19:36.222 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:19:36.222 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=22385 2024-09-13T05:19:36.222 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:36.222 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:36.222 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:36.222 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:36.223 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:36.225 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:36.225 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:36.225 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:36.296 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:36.294+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:36.308 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:36.308 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:36.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:36.308 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:36.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.308 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.308 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.977873309, 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:19:36.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.314 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.966667548, 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:36.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:36.319 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.956488494, 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:19:36.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.949052691, 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:19:36.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:19:36.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.329 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.942444988, 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:19:36.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:19:36.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:19:36.335 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.93321877, 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:19:36.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:19:36.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4797.0:1 3.3 e48/47 3:c4c92e5a:::rbd_trash:head v 48'1, mlcod=0'0)", 2024-09-13T05:19:36.340 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.845724856, 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015947019999999999, 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459810+0000", 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8649999999999998e-06 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459820+0000", 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9899999999999992e-06 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459829+0000", 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.431e-06 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459885+0000", 2024-09-13T05:19:36.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5868000000000001e-05 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459912+0000", 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7203e-05 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461364+0000", 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014515820000000001 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461399+0000", 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5063e-05 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4664.0:119 2.3 e50/16 2:d073633c:::rbd_header.11d2ef348a4:head v 50'39, mlcod=50'37)", 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:26.364298+0000", 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9412315430000007, 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013753039999999999, 2024-09-13T05:19:36.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364298+0000", 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364298+0000", 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364296+0000", 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364309+0000", 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3246e-05 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364340+0000", 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0216e-05 2024-09-13T05:19:36.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364352+0000", 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2327000000000001e-05 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364454+0000", 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010179399999999999 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364497+0000", 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2916e-05 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.365601+0000", 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001104397 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.365673+0000", 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2047999999999999e-05 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.349 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4679.0:12 2.3 e50/16 2:d073633c:::rbd_header.11d2ef348a4:head v 50'40, mlcod=50'38)", 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.016450+0000", 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.2890793509999998, 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001286354, 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016450+0000", 2024-09-13T05:19:36.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016450+0000", 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016447+0000", 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016472+0000", 2024-09-13T05:19:36.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5015999999999999e-05 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016522+0000", 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9549000000000003e-05 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016545+0000", 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3275999999999999e-05 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016658+0000", 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112667 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.016698+0000", 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0626e-05 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.017663+0000", 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00096442499999999998 2024-09-13T05:19:36.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.017736+0000", 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3480999999999997e-05 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:180 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8811787249999998, 2024-09-13T05:19:36.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013961119999999999, 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 180 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424350+0000", 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424354+0000", 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3560000000000003e-06 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424365+0000", 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0832999999999999e-05 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424374+0000", 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5599999999999994e-06 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424997+0000", 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00062386799999999997 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.425131+0000", 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133614 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.425747+0000", 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00061550899999999998 2024-09-13T05:19:36.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:494 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.856467+0000", 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.4490621610000005, 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001401575, 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:36.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 494 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856467+0000", 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856467+0000", 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856466+0000", 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856473+0000", 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4839999999999998e-06 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856493+0000", 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9819000000000001e-05 2024-09-13T05:19:36.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856509+0000", 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5333e-05 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.857232+0000", 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00072378299999999998 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.857354+0000", 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121625 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.857869+0000", 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00051458799999999996 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.4385093700000002, 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:36.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:19:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:510 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.878248+0000", 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.4272814399999998, 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001141258, 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:36.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 510 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.878248+0000", 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.878248+0000", 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.878247+0000", 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.878252+0000", 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.279e-06 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.878262+0000", 2024-09-13T05:19:36.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295000000000001e-05 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.878272+0000", 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031e-05 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.878682+0000", 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040994600000000001 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.878857+0000", 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017473599999999999 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.879389+0000", 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053224100000000003 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:518 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:36.372 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.888353+0000", 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.4171759720000008, 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013167999999999999, 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 518 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888353+0000", 2024-09-13T05:19:36.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888353+0000", 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888353+0000", 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888355+0000", 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3819999999999998e-06 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888360+0000", 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0849999999999996e-06 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888364+0000", 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8519999999999997e-06 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888978+0000", 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00061398300000000002 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.889137+0000", 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015835299999999999 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.889670+0000", 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053354100000000001 2024-09-13T05:19:36.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:527 2.0 2:0158b2c3:::rbd_data.11d2ef348a4.000000000000000e:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.4058488820000008, 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001391878, 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:36.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 527 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899682+0000", 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.198e-06 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899687+0000", 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8760000000000004e-06 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899691+0000", 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6509999999999999e-06 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.900302+0000", 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00061118999999999995 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.900505+0000", 2024-09-13T05:19:36.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000203523 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.901072+0000", 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00056678000000000002 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:543 2.0 2:0158b2c3:::rbd_data.11d2ef348a4.000000000000000e:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.920384+0000", 2024-09-13T05:19:36.381 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.3851454259999993, 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001267811, 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 543 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.920384+0000", 2024-09-13T05:19:36.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.920384+0000", 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.920383+0000", 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.920388+0000", 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5639999999999998e-06 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.920423+0000", 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5692000000000003e-05 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.920431+0000", 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0800000000000006e-06 2024-09-13T05:19:36.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.920934+0000", 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050236200000000001 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.921103+0000", 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016925600000000001 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.921652+0000", 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00054856100000000004 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:551 2.0 2:0158b2c3:::rbd_data.11d2ef348a4.000000000000000e:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.931321+0000", 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.3742081230000007, 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001380287, 2024-09-13T05:19:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 551 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931321+0000", 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931321+0000", 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931321+0000", 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931323+0000", 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.633e-06 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931328+0000", 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0830000000000003e-06 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931332+0000", 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7529999999999999e-06 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931903+0000", 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000571296 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.932067+0000", 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016321 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.932701+0000", 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00063475200000000004 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 51/16 rep_tid 236 entries 51'57 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.4917.0:78 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 51'55)", 2024-09-13T05:19:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:30.644268+0000", 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.6612610349999999, 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001243086, 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.644268+0000", 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.644268+0000", 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.644266+0000", 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.644289+0000", 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2663e-05 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.644330+0000", 2024-09-13T05:19:36.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0788999999999999e-05 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.644374+0000", 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4005999999999998e-05 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.644544+0000", 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017051300000000001 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.645511+0000", 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00096711500000000005 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 51/16 rep_tid 237 entries 51'58 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.4917.0:96 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 51'56)", 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:30.660637+0000", 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.6448920940000002, 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012409159999999999, 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.660637+0000", 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.660637+0000", 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.660635+0000", 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.660664+0000", 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8316000000000001e-05 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.660716+0000", 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1965999999999997e-05 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.660741+0000", 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5562e-05 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.660886+0000", 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014443400000000001 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:30.661878+0000", 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099251200000000008 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4784.0:18 2.3 e54/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 54'104, mlcod=54'102)", 2024-09-13T05:19:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:34.977408+0000", 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.3281211420000001, 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00129685, 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.977408+0000", 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.977408+0000", 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.977407+0000", 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.977414+0000", 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9940000000000003e-06 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.977429+0000", 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4714e-05 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.977436+0000", 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1060000000000002e-06 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.977539+0000", 2024-09-13T05:19:36.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010279 2024-09-13T05:19:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.977574+0000", 2024-09-13T05:19:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5026e-05 2024-09-13T05:19:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.978675+0000", 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001100972 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:34.978705+0000", 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0060999999999999e-05 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:36.402 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:36.405 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:36.405 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:36.405 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^22385\b' 2024-09-13T05:19:36.405 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^22385\b' 2024-09-13T05:19:36.405 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:36.406 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:36.406 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *22385\b' 2024-09-13T05:19:36.406 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *22385\b' 2024-09-13T05:19:36.406 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:36.406 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:36.406 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:36.401+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:36.505 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:36.502+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:36.537 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:36.537 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:36.537 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:36.551 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:36.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:36.606+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:36.666 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:36.666 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:36.666 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:36.666 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:36.666 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:36.680 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:36.727 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:36.725+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:36.803 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:36.803 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:36.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:36.803 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:36.803 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.803 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:36.803 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.668612725000003, 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.667981466000001, 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:36.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:36.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:36.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.667336344999995, 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.812 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:36.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:36.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.666828981999998, 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:36.816 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:36.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.666303545999995, 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.665666033999997, 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:36.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.829 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.665089326, 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:36.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.664537852999999, 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.834 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:36.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:36.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.838 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.664077270000007, 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:36.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:36.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:36.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.663567384000004, 2024-09-13T05:19:36.843 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.844 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:36.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:36.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.847 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.663140321, 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.848 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:36.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:36.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:36.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.276380603000007, 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.852 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:36.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:36.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.856 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 85.658845791000005, 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.857 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:36.858 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:36.859 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.860 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.428066855999999, 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.861 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:36.862 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:19:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.865 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.417446512, 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:36.866 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.410232197999999, 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.871 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:36.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:19:36.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:19:36.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:19:36.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.403876678, 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:36.877 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:19:36.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:19:36.880 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:36.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.882 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.403820314000001, 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.883 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:19:36.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:19:36.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:36.886 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.887 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.888 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.403782743000001, 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.891 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:19:36.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:19:36.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.403748219000001, 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.895 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:19:36.896 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:36.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:36.899 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:36.900 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:19:36.900 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:19:36.900 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.900 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.900 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.900 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:36.900 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:36.900 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:36.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:36.903 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:36.904 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:36.897+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:36.904 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd --device-type nbd map --exclusive rbd/testrbdnbd21435 2024-09-13T05:19:36.904 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:36.904 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:36.904 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:36.904 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:36.904 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:36.904 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:36.904 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --exclusive rbd/testrbdnbd21435 2024-09-13T05:19:37.002 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:37.002+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:37.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:37.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:37.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:37.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:37.045 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:37.045 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:37.060 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:37.073 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:37.073 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:37.074 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:37.074 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:37.076 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:37.076 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:37.079 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:19:37.079 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T05:19:37.089 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:37.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T05:19:37.106 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:37.107 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:37.106+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:37.117 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=22457 2024-09-13T05:19:37.117 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 22457 2024-09-13T05:19:37.117 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 22457 -C rbd-nbd 2024-09-13T05:19:37.124 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:37.124 INFO:tasks.workunit.client.0.smithi122.stdout: 22457 ? 00:00:00 rbd-nbd 2024-09-13T05:19:37.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:37.125 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:37.125 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-13T05:19:37.125 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:37.125 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:37.127 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:37.127 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:19:37.129 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:19:37.129 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:37.129 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-13T05:19:37.144 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:37.165 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:37.165 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:37.215 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:37.215 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:37.230 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:37.230+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:37.231 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:37.304 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:37.302+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:37.360 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:37.360 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:37.378 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:37.423 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:37.421+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:37.498 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:37.499 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:37.505 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:37.501+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:37.515 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:37.606 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:37.606+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:37.686 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:37.687 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:37.701 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:37.730 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records in 2024-09-13T05:19:37.730 INFO:tasks.workunit.client.0.smithi122.stderr:64+0 records out 2024-09-13T05:19:37.730 INFO:tasks.workunit.client.0.smithi122.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.594806 s, 113 MB/s 2024-09-13T05:19:37.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false timeout 10 rbd bench testrbdnbd21435 --io-type write --io-size=1024 --io-total=1024 2024-09-13T05:19:37.732 INFO:tasks.workunit.client.0.smithi122.stderr:+ timeout 10 rbd bench testrbdnbd21435 --io-type write --io-size=1024 --io-total=1024 2024-09-13T05:19:37.747 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:37.745+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:37.808 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:37.806+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:37.808 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: failed to flush: 2024-09-13T05:19:37.807+0000 7f05289c4640 -1 librbd::exclusive_lock::ImageDispatch: 0x7f050c000ff0 handle_acquire_lock: failed to acquire exclusive lock: (30) Read-only file system 2024-09-13T05:19:37.808 INFO:tasks.workunit.client.0.smithi122.stderr:(30) Read-only file system2024-09-13T05:19:37.807+0000 7f05289c4640 -1 librbd::io::AioCompletion: 0x55ce838ca1c0 fail: (30) Read-only file system 2024-09-13T05:19:37.809 INFO:tasks.workunit.client.0.smithi122.stderr: 2024-09-13T05:19:37.809 INFO:tasks.workunit.client.0.smithi122.stderr:bench failed: (30) Read-only file system 2024-09-13T05:19:37.817 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:37.817 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 22457 2024-09-13T05:19:37.817 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:19:37.817 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=22457 2024-09-13T05:19:37.817 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:37.817 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:37.817 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:37.818 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:37.818 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:37.819 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:37.820 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:37.820 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:37.821 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:37.822 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:37.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:37.903 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:37.904 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^22457\b' 2024-09-13T05:19:37.904 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^22457\b' 2024-09-13T05:19:37.908 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:37.905+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:37.950 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:37.951 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:37.951 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *22457\b' 2024-09-13T05:19:37.951 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *22457\b' 2024-09-13T05:19:37.957 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:37.957 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:37.971 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:37.971 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:37.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:38.030 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:38.030+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T05:19:38.109 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:38.110+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:38.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T05:19:38.112 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:38.112 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:38.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:38.210 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:38.209+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:38.302 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:38.302 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:38.312 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:38.309+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:38.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:38.413 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:38.413+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:38.435 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:38.435 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:38.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:38.459 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:38.459 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd bench testrbdnbd21435 --io-type write --io-size=1024 --io-total=1024 2024-09-13T05:19:38.513 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:38.509+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:38.523 INFO:tasks.workunit.client.0.smithi122.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T05:19:38.525 INFO:tasks.workunit.client.0.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T05:19:38.525 INFO:tasks.workunit.client.0.smithi122.stdout:elapsed: 0 ops: 1 ops/sec: 250.011 bytes/sec: 250 KiB/s 2024-09-13T05:19:38.536 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:38.536 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:38.536 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:38.537 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:38.537 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:38.538 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:38.538 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:38.538 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:38.577 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:38.578 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:38.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:38.638 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:38.638+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:38.659 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:38.659 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:38.659 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:38.660 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:38.660 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:38.660 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:38.715 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:38.713+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:38.721 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:38.722 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:38.726 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=22571 2024-09-13T05:19:38.726 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 22571 2024-09-13T05:19:38.726 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 22571 -C rbd-nbd 2024-09-13T05:19:38.733 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:38.733 INFO:tasks.workunit.client.0.smithi122.stdout: 22571 ? 00:00:00 rbd-nbd 2024-09-13T05:19:38.733 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device testrbdnbd21435 22571 2024-09-13T05:19:38.734 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=testrbdnbd21435 2024-09-13T05:19:38.734 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=22571 2024-09-13T05:19:38.734 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd21435 2024-09-13T05:19:38.734 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:38.734 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:38.734 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:38.735 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:38.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:38.737 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:38.737 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:38.737 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd21435 2024-09-13T05:19:38.821 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:38.822 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^22571\b' 2024-09-13T05:19:38.822 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^22571\b' 2024-09-13T05:19:38.822 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:38.822+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:38.868 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:38.869 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:38.869 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *22571\b' 2024-09-13T05:19:38.869 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *22571\b' 2024-09-13T05:19:38.876 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:38.876 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:38.906 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:38.906 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:38.917 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:38.914+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:38.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:39.018 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:39.013+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:39.086 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:39.086 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:39.092 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T05:19:39.104 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:39.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:19:39.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T05:19:39.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:39.113+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:39.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:39.219 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:39.217+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:39.232 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:39.232 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:39.249 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:39.319 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:39.317+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:39.357 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:39.357 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:39.373 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:39.377 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:39.377 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd21435@snap 2024-09-13T05:19:39.381 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:39.381 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:39.391 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:39.391 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:39.398 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:39.411 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:39.411 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:39.420 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:39.418+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:39.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:39.429 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:39.450 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:39.450 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:39.521 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:39.517+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:39.564 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:39.564 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:39.564 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:39.564 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:39.564 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:39.578 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:39.623 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:39.622+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:39.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:39.722+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:39.742 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.608578813999998, 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.743 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:39.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.745 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.746 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:39.747 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.607947554999996, 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:39.751 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.607302434000005, 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.752 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:39.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.606795070999993, 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.758 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.758 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:39.758 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.758 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.758 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.758 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.758 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:39.758 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.758 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:39.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:39.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.762 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:39.762 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:39.762 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.606269635000004, 2024-09-13T05:19:39.762 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:39.762 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.762 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.762 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:39.764 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:39.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.765 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:39.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:39.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.765 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:39.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.605632123000007, 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:39.768 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.769 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.770 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.771 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.605055414999995, 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:39.772 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.773 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.604503941999994, 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.777 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.778 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.781 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.604043359000002, 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.782 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.783 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.784 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:39.785 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.603533472999999, 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.786 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.787 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:39.788 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:39.789 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.790 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.603106409999995, 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.791 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.792 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.793 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.794 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.795 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.216346692000002, 2024-09-13T05:19:39.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.797 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.798 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.59881188, 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.801 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.802 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.802 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:39.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.802 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.802 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:39.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:39.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:39.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:39.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:39.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:39.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:39.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:39.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.368032945, 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:39.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:19:39.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:19:39.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.357412601, 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:19:39.811 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:19:39.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:39.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.816 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.350198287000001, 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:39.817 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:39.818 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:39.818 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:19:39.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.818 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.818 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:39.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:19:39.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:19:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:39.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.343842767000002, 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:39.825 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:19:39.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:19:39.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:19:39.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.343786402999999, 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:19:39.830 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.831 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.831 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:39.831 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:39.831 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:39.831 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:19:39.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.831 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.832 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:19:39.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:39.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:19:39.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:19:39.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:39.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:19:39.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.837 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:19:39.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:19:39.837 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.837 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.837 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.837 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:39.837 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.343748831999999, 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.838 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:19:39.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:19:39.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.841 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:19:39.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:19:39.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.919 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.343714307999999, 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:39.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:39.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:39.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:39.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:19:39.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:19:39.925 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.925 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.925 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.925 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:39.925 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:39.925 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:39.927 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:39.927 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:39.926+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:40.028 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:40.025+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:40.058 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:40.058 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:40.059 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:40.059 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:40.073 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:40.103 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T05:19:40.104 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:19:40.119 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T05:19:40.129 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:40.126+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:40.133 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T05:19:40.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:19:40.134 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21435@snap 2024-09-13T05:19:40.134 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:40.135 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:40.136 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:40.137 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:40.139 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:40.139 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:40.139 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21435@snap 2024-09-13T05:19:40.230 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:40.230+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:40.238 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:40.238 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:40.238 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:40.238 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:40.238 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:40.252 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:40.279 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:40.279 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:40.279 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:40.279 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:40.280 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:40.280 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:40.331 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:40.329+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:40.343 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=22668 2024-09-13T05:19:40.343 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 22668 2024-09-13T05:19:40.343 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 22668 -C rbd-nbd 2024-09-13T05:19:40.351 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:40.351 INFO:tasks.workunit.client.0.smithi122.stdout: 22668 ? 00:00:00 rbd-nbd 2024-09-13T05:19:40.352 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device testrbdnbd21435@snap 22668 2024-09-13T05:19:40.352 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=testrbdnbd21435@snap 2024-09-13T05:19:40.352 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=22668 2024-09-13T05:19:40.352 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd21435@snap 2024-09-13T05:19:40.352 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:40.352 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:40.353 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:40.353 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:40.354 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:40.354 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:40.354 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd21435@snap 2024-09-13T05:19:40.376 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:40.376 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:40.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:40.376 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:40.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.376 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:40.376 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:40.376 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 19.03538292, 2024-09-13T05:19:40.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.378 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:19:40.381 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 19.023502317999998, 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.383 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:19:40.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:19:40.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 19.017558618999999, 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.386 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:19:40.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.389 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 19.014882911000001, 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.394 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 19.010168878000002, 2024-09-13T05:19:40.396 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.400 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:19:40.400 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:19:40.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.400 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.400 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:19:40.400 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:19:40.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.994216604999998, 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.403 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:19:40.405 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.984576402999998, 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.977483597999999, 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:19:40.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:19:40.414 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.969265642, 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:40.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:19:40.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.967119829000001, 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.420 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:19:40.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.423 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.424 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:19:40.424 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:19:40.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.424 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.424 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.424 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:19:40.424 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:19:40.424 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.424 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.965202035000001, 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.425 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:40.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.426 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.426 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:40.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.426 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:19:40.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:19:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:19:40.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:19:40.434 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.434 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.434 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.434 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:56 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'24, mlcod=46'17)", 2024-09-13T05:19:40.434 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.412806+0000", 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.963431290999999, 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001612657, 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412806+0000", 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.435 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412806+0000", 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412804+0000", 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412819+0000", 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4842e-05 2024-09-13T05:19:40.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412847+0000", 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7458999999999999e-05 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412864+0000", 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7141000000000001e-05 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413020+0000", 2024-09-13T05:19:40.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000156346 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413060+0000", 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9774999999999998e-05 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414357+0000", 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0012974569999999999 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414418+0000", 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1062999999999995e-05 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 12.811755773, 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.440 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:40.440 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:40.440 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:19:40.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.440 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.440 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:19:40.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 12.805317833, 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:19:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.447 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:19:40.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:19:40.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.447 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.447 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:19:40.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:19:40.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:19:40.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 12.799249679999999, 2024-09-13T05:19:40.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.453 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 12.791706126999999, 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.455 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.457 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 12.779028882, 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.458 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.461 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 4.292405016, 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:40.462 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:19:40.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 4.2923361949999999, 2024-09-13T05:19:40.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:19:40.468 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.468 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:40.468 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.468 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:19:40.468 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:40.468 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:19:40.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.468 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:40.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:19:40.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:19:40.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:19:40.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:19:40.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:19:40.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:19:40.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:19:40.473 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.473 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.473 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.26393362500000001, 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.474 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:19:40.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:19:40.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:19:40.480 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.480 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.480 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.480 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.480 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.480 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:40.481 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:40.481 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^22668\b' 2024-09-13T05:19:40.481 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^22668\b' 2024-09-13T05:19:40.482 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:40.483 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:40.482+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:19:40.484 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:19:40.489 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:40.489 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:40.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *22668\b' 2024-09-13T05:19:40.490 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *22668\b' 2024-09-13T05:19:40.496 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:40.496 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:40.501 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-13T05:19:40.100598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":130613937,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2421,"num_read_kb":160346,"num_write":685,"num_write_kb":260642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":607,"ondisk_log_size":607,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":418108,"kb_used_data":232432,"kb_used_omap":14,"kb_used_meta":184689,"kb_avail":565812932,"statfs":{"total":579820584960,"available":579392442368,"internally_reserved":0,"allocated":238010368,"data_stored":235536476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":189121654},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4194304,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":828,"num_read_kb":2699,"num_write":264,"num_write_kb":61540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29360128,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030227"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":200,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T05:19:36.099962+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:19:36.099962+0000","last_peered":"2024-09-13T05:19:36.099962+0000","last_clean":"2024-09-13T05:19:36.099962+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:19:36.099962+0000","last_undegraded":"2024-09-13T05:19:36.099962+0000","last_fullsized":"2024-09-13T05:19:36.099962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"57'154","reported_seq":750,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.893200+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:19:37.893200+0000","last_peered":"2024-09-13T05:19:37.893200+0000","last_clean":"2024-09-13T05:19:37.893200+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:19:37.893200+0000","last_undegraded":"2024-09-13T05:19:37.893200+0000","last_fullsized":"2024-09-13T05:19:37.893200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":12582950,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1345,"num_read_kb":12748,"num_write":178,"num_write_kb":20614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"50'2","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T05:19:35.069785+0000","last_change":"2024-09-13T05:19:25.480800+0000","last_active":"2024-09-13T05:19:35.069785+0000","last_peered":"2024-09-13T05:19:35.069785+0000","last_clean":"2024-09-13T05:19:35.069785+0000","last_became_active":"2024-09-13T05:19:23.455354+0000","last_became_peered":"2024-09-13T05:19:23.455354+0000","last_unstale":"2024-09-13T05:19:35.069785+0000","last_undegraded":"2024-09-13T05:19:35.069785+0000","last_fullsized":"2024-09-13T05:19:35.069785+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:20.817288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017913759999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"57'58","reported_seq":213,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.681466+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:19:37.681466+0000","last_peered":"2024-09-13T05:19:37.681466+0000","last_clean":"2024-09-13T05:19:37.681466+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:19:37.681466+0000","last_undegraded":"2024-09-13T05:19:37.681466+0000","last_fullsized":"2024-09-13T05:19:37.681466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":25165824,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":102,"num_read_kb":24576,"num_write":57,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137737+0000","last_change":"2024-09-13T05:19:35.075887+0000","last_active":"2024-09-13T05:19:37.137737+0000","last_peered":"2024-09-13T05:19:37.137737+0000","last_clean":"2024-09-13T05:19:37.137737+0000","last_became_active":"2024-09-13T05:19:35.073898+0000","last_became_peered":"2024-09-13T05:19:35.073898+0000","last_unstale":"2024-09-13T05:19:37.137737+0000","last_undegraded":"2024-09-13T05:19:37.137737+0000","last_fullsized":"2024-09-13T05:19:37.137737+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:23.211903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010913139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"53'12","reported_seq":102,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T05:19:35.069785+0000","last_change":"2024-09-13T05:18:43.272711+0000","last_active":"2024-09-13T05:19:35.069785+0000","last_peered":"2024-09-13T05:19:35.069785+0000","last_clean":"2024-09-13T05:19:35.069785+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:19:35.069785+0000","last_undegraded":"2024-09-13T05:19:35.069785+0000","last_fullsized":"2024-09-13T05:19:35.069785+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:27.589949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194320,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":8193,"num_write":12,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137875+0000","last_change":"2024-09-13T05:19:25.538085+0000","last_active":"2024-09-13T05:19:37.137875+0000","last_peered":"2024-09-13T05:19:37.137875+0000","last_clean":"2024-09-13T05:19:37.137875+0000","last_became_active":"2024-09-13T05:19:23.462578+0000","last_became_peered":"2024-09-13T05:19:23.462578+0000","last_unstale":"2024-09-13T05:19:37.137875+0000","last_undegraded":"2024-09-13T05:19:37.137875+0000","last_fullsized":"2024-09-13T05:19:37.137875+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:48.193321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056272899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"57'56","reported_seq":285,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.765965+0000","last_change":"2024-09-13T05:19:24.517596+0000","last_active":"2024-09-13T05:19:37.765965+0000","last_peered":"2024-09-13T05:19:37.765965+0000","last_clean":"2024-09-13T05:19:37.765965+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:19:37.765965+0000","last_undegraded":"2024-09-13T05:19:37.765965+0000","last_fullsized":"2024-09-13T05:19:37.765965+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":153,"num_read_kb":36933,"num_write":56,"num_write_kb":36866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"48'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T05:19:36.070614+0000","last_change":"2024-09-13T05:19:25.535626+0000","last_active":"2024-09-13T05:19:36.070614+0000","last_peered":"2024-09-13T05:19:36.070614+0000","last_clean":"2024-09-13T05:19:36.070614+0000","last_became_active":"2024-09-13T05:19:23.458938+0000","last_became_peered":"2024-09-13T05:19:23.458938+0000","last_unstale":"2024-09-13T05:19:36.070614+0000","last_undegraded":"2024-09-13T05:19:36.070614+0000","last_fullsized":"2024-09-13T05:19:36.070614+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:28:08.756172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000291315,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.136618+0000","last_change":"2024-09-13T05:19:25.474324+0000","last_active":"2024-09-13T05:19:37.136618+0000","last_peered":"2024-09-13T05:19:37.136618+0000","last_clean":"2024-09-13T05:19:37.136618+0000","last_became_active":"2024-09-13T05:19:23.455224+0000","last_became_peered":"2024-09-13T05:19:23.455224+0000","last_unstale":"2024-09-13T05:19:37.136618+0000","last_undegraded":"2024-09-13T05:19:37.136618+0000","last_fullsized":"2024-09-13T05:19:37.136618+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:43.322756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"57'59","reported_seq":211,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.588061+0000","last_change":"2024-09-13T05:18:43.275214+0000","last_active":"2024-09-13T05:19:37.588061+0000","last_peered":"2024-09-13T05:19:37.588061+0000","last_clean":"2024-09-13T05:19:37.588061+0000","last_became_active":"2024-09-13T05:18:43.274886+0000","last_became_peered":"2024-09-13T05:18:43.274886+0000","last_unstale":"2024-09-13T05:19:37.588061+0000","last_undegraded":"2024-09-13T05:19:37.588061+0000","last_fullsized":"2024-09-13T05:19:37.588061+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:53.129994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":16777254,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":188,"num_read_kb":16477,"num_write":75,"num_write_kb":32803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137738+0000","last_change":"2024-09-13T05:19:35.163055+0000","last_active":"2024-09-13T05:19:37.137738+0000","last_peered":"2024-09-13T05:19:37.137738+0000","last_clean":"2024-09-13T05:19:37.137738+0000","last_became_active":"2024-09-13T05:19:35.161280+0000","last_became_peered":"2024-09-13T05:19:35.161280+0000","last_unstale":"2024-09-13T05:19:37.137738+0000","last_undegraded":"2024-09-13T05:19:37.137738+0000","last_fullsized":"2024-09-13T05:19:37.137738+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T05:49:15.311928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092053199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"57'40","reported_seq":187,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.289965+0000","last_change":"2024-09-13T05:19:35.195473+0000","last_active":"2024-09-13T05:19:37.289965+0000","last_peered":"2024-09-13T05:19:37.289965+0000","last_clean":"2024-09-13T05:19:37.289965+0000","last_became_active":"2024-09-13T05:19:35.077291+0000","last_became_peered":"2024-09-13T05:19:35.077291+0000","last_unstale":"2024-09-13T05:19:37.289965+0000","last_undegraded":"2024-09-13T05:19:37.289965+0000","last_fullsized":"2024-09-13T05:19:37.289965+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:50.348812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":16384,"num_write":40,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137908+0000","last_change":"2024-09-13T05:19:35.076046+0000","last_active":"2024-09-13T05:19:37.137908+0000","last_peered":"2024-09-13T05:19:37.137908+0000","last_clean":"2024-09-13T05:19:37.137908+0000","last_became_active":"2024-09-13T05:19:35.074375+0000","last_became_peered":"2024-09-13T05:19:35.074375+0000","last_unstale":"2024-09-13T05:19:37.137908+0000","last_undegraded":"2024-09-13T05:19:37.137908+0000","last_fullsized":"2024-09-13T05:19:37.137908+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:58.769823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089178300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"57'28","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.510695+0000","last_change":"2024-09-13T05:19:35.179253+0000","last_active":"2024-09-13T05:19:37.510695+0000","last_peered":"2024-09-13T05:19:37.510695+0000","last_clean":"2024-09-13T05:19:37.510695+0000","last_became_active":"2024-09-13T05:19:35.076916+0000","last_became_peered":"2024-09-13T05:19:35.076916+0000","last_unstale":"2024-09-13T05:19:37.510695+0000","last_undegraded":"2024-09-13T05:19:37.510695+0000","last_fullsized":"2024-09-13T05:19:37.510695+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:40:25.275387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":12315,"num_write":26,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137789+0000","last_change":"2024-09-13T05:19:35.078422+0000","last_active":"2024-09-13T05:19:37.137789+0000","last_peered":"2024-09-13T05:19:37.137789+0000","last_clean":"2024-09-13T05:19:37.137789+0000","last_became_active":"2024-09-13T05:19:35.076279+0000","last_became_peered":"2024-09-13T05:19:35.076279+0000","last_unstale":"2024-09-13T05:19:37.137789+0000","last_undegraded":"2024-09-13T05:19:37.137789+0000","last_fullsized":"2024-09-13T05:19:37.137789+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:47.540962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012842699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"57'80","reported_seq":448,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.783130+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:19:37.783130+0000","last_peered":"2024-09-13T05:19:37.783130+0000","last_clean":"2024-09-13T05:19:37.783130+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:19:37.783130+0000","last_undegraded":"2024-09-13T05:19:37.783130+0000","last_fullsized":"2024-09-13T05:19:37.783130+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":16777231,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":342,"num_read_kb":32620,"num_write":80,"num_write_kb":40961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":130023550,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2305,"num_read_kb":160246,"num_write":524,"num_write_kb":258224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":255893504,"data_stored":255852796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":487,"ondisk_log_size":487,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607575,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71244,"kb_used_data":43332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94300596,"statfs":{"total":96636764160,"available":96563810304,"internally_reserved":0,"allocated":44371968,"data_stored":43954863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.72299999999999998}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.70199999999999996}]},{"osd":2,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.66100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]}]},{"osd":4,"up_from":54,"seq":231928233986,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69704,"kb_used_data":41592,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94302136,"statfs":{"total":96636764160,"available":96565387264,"internally_reserved":0,"allocated":42590208,"data_stored":42184793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]}]},{"osd":3,"up_from":29,"seq":124554051596,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41020,"kb_used_data":12908,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94330820,"statfs":{"total":96636764160,"available":96594759680,"internally_reserved":0,"allocated":13217792,"data_stored":12814726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607574,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72712,"kb_used_data":37504,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94299128,"statfs":{"total":96636764160,"available":96562307072,"internally_reserved":0,"allocated":38404096,"data_stored":37990546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.51200000000000001}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.625},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.55400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":5,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.871},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.78200000000000003}]}]},{"osd":1,"up_from":12,"seq":51539607574,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101968,"kb_used_data":66760,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94269872,"statfs":{"total":96636764160,"available":96532348928,"internally_reserved":0,"allocated":68362240,"data_stored":67941057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.41999999999999998}]},{"osd":2,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":5,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.625}]}]},{"osd":0,"up_from":12,"seq":51539607574,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61460,"kb_used_data":30336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94310380,"statfs":{"total":96636764160,"available":96573829120,"internally_reserved":0,"allocated":31064064,"data_stored":30650491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":5,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.86199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46145536,"data_stored":46137378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67121152,"data_stored":67108936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37752832,"data_stored":37748774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41943040,"data_stored":41943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50331702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:19:40.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:19:40.590 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:40.590+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:40.650 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:40.650 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:40.650 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:40.650 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:40.664 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:40.711 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:40.709+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:40.781 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:40.781 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:40.781 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:40.781 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:40.781 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:40.783 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:40.782+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:40.796 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:19:40.850 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:19:40.869 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-13T05:19:40.100598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":130613937,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2421,"num_read_kb":160346,"num_write":685,"num_write_kb":260642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":607,"ondisk_log_size":607,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":418108,"kb_used_data":232432,"kb_used_omap":14,"kb_used_meta":184689,"kb_avail":565812932,"statfs":{"total":579820584960,"available":579392442368,"internally_reserved":0,"allocated":238010368,"data_stored":235536476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":189121654},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4194304,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":828,"num_read_kb":2699,"num_write":264,"num_write_kb":61540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29360128,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030227"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":200,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T05:19:36.099962+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:19:36.099962+0000","last_peered":"2024-09-13T05:19:36.099962+0000","last_clean":"2024-09-13T05:19:36.099962+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:19:36.099962+0000","last_undegraded":"2024-09-13T05:19:36.099962+0000","last_fullsized":"2024-09-13T05:19:36.099962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"57'154","reported_seq":750,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.893200+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:19:37.893200+0000","last_peered":"2024-09-13T05:19:37.893200+0000","last_clean":"2024-09-13T05:19:37.893200+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:19:37.893200+0000","last_undegraded":"2024-09-13T05:19:37.893200+0000","last_fullsized":"2024-09-13T05:19:37.893200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":12582950,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1345,"num_read_kb":12748,"num_write":178,"num_write_kb":20614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"50'2","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T05:19:35.069785+0000","last_change":"2024-09-13T05:19:25.480800+0000","last_active":"2024-09-13T05:19:35.069785+0000","last_peered":"2024-09-13T05:19:35.069785+0000","last_clean":"2024-09-13T05:19:35.069785+0000","last_became_active":"2024-09-13T05:19:23.455354+0000","last_became_peered":"2024-09-13T05:19:23.455354+0000","last_unstale":"2024-09-13T05:19:35.069785+0000","last_undegraded":"2024-09-13T05:19:35.069785+0000","last_fullsized":"2024-09-13T05:19:35.069785+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:20.817288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017913759999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"57'58","reported_seq":213,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.681466+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:19:37.681466+0000","last_peered":"2024-09-13T05:19:37.681466+0000","last_clean":"2024-09-13T05:19:37.681466+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:19:37.681466+0000","last_undegraded":"2024-09-13T05:19:37.681466+0000","last_fullsized":"2024-09-13T05:19:37.681466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":25165824,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":102,"num_read_kb":24576,"num_write":57,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137737+0000","last_change":"2024-09-13T05:19:35.075887+0000","last_active":"2024-09-13T05:19:37.137737+0000","last_peered":"2024-09-13T05:19:37.137737+0000","last_clean":"2024-09-13T05:19:37.137737+0000","last_became_active":"2024-09-13T05:19:35.073898+0000","last_became_peered":"2024-09-13T05:19:35.073898+0000","last_unstale":"2024-09-13T05:19:37.137737+0000","last_undegraded":"2024-09-13T05:19:37.137737+0000","last_fullsized":"2024-09-13T05:19:37.137737+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:23.211903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010913139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"53'12","reported_seq":102,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-13T05:19:35.069785+0000","last_change":"2024-09-13T05:18:43.272711+0000","last_active":"2024-09-13T05:19:35.069785+0000","last_peered":"2024-09-13T05:19:35.069785+0000","last_clean":"2024-09-13T05:19:35.069785+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:19:35.069785+0000","last_undegraded":"2024-09-13T05:19:35.069785+0000","last_fullsized":"2024-09-13T05:19:35.069785+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:27.589949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194320,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":8193,"num_write":12,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137875+0000","last_change":"2024-09-13T05:19:25.538085+0000","last_active":"2024-09-13T05:19:37.137875+0000","last_peered":"2024-09-13T05:19:37.137875+0000","last_clean":"2024-09-13T05:19:37.137875+0000","last_became_active":"2024-09-13T05:19:23.462578+0000","last_became_peered":"2024-09-13T05:19:23.462578+0000","last_unstale":"2024-09-13T05:19:37.137875+0000","last_undegraded":"2024-09-13T05:19:37.137875+0000","last_fullsized":"2024-09-13T05:19:37.137875+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:48.193321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056272899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"57'56","reported_seq":285,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.765965+0000","last_change":"2024-09-13T05:19:24.517596+0000","last_active":"2024-09-13T05:19:37.765965+0000","last_peered":"2024-09-13T05:19:37.765965+0000","last_clean":"2024-09-13T05:19:37.765965+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:19:37.765965+0000","last_undegraded":"2024-09-13T05:19:37.765965+0000","last_fullsized":"2024-09-13T05:19:37.765965+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":153,"num_read_kb":36933,"num_write":56,"num_write_kb":36866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"48'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-13T05:19:36.070614+0000","last_change":"2024-09-13T05:19:25.535626+0000","last_active":"2024-09-13T05:19:36.070614+0000","last_peered":"2024-09-13T05:19:36.070614+0000","last_clean":"2024-09-13T05:19:36.070614+0000","last_became_active":"2024-09-13T05:19:23.458938+0000","last_became_peered":"2024-09-13T05:19:23.458938+0000","last_unstale":"2024-09-13T05:19:36.070614+0000","last_undegraded":"2024-09-13T05:19:36.070614+0000","last_fullsized":"2024-09-13T05:19:36.070614+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:28:08.756172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000291315,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.136618+0000","last_change":"2024-09-13T05:19:25.474324+0000","last_active":"2024-09-13T05:19:37.136618+0000","last_peered":"2024-09-13T05:19:37.136618+0000","last_clean":"2024-09-13T05:19:37.136618+0000","last_became_active":"2024-09-13T05:19:23.455224+0000","last_became_peered":"2024-09-13T05:19:23.455224+0000","last_unstale":"2024-09-13T05:19:37.136618+0000","last_undegraded":"2024-09-13T05:19:37.136618+0000","last_fullsized":"2024-09-13T05:19:37.136618+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:43.322756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"57'59","reported_seq":211,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.588061+0000","last_change":"2024-09-13T05:18:43.275214+0000","last_active":"2024-09-13T05:19:37.588061+0000","last_peered":"2024-09-13T05:19:37.588061+0000","last_clean":"2024-09-13T05:19:37.588061+0000","last_became_active":"2024-09-13T05:18:43.274886+0000","last_became_peered":"2024-09-13T05:18:43.274886+0000","last_unstale":"2024-09-13T05:19:37.588061+0000","last_undegraded":"2024-09-13T05:19:37.588061+0000","last_fullsized":"2024-09-13T05:19:37.588061+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:53.129994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":16777254,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":188,"num_read_kb":16477,"num_write":75,"num_write_kb":32803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137738+0000","last_change":"2024-09-13T05:19:35.163055+0000","last_active":"2024-09-13T05:19:37.137738+0000","last_peered":"2024-09-13T05:19:37.137738+0000","last_clean":"2024-09-13T05:19:37.137738+0000","last_became_active":"2024-09-13T05:19:35.161280+0000","last_became_peered":"2024-09-13T05:19:35.161280+0000","last_unstale":"2024-09-13T05:19:37.137738+0000","last_undegraded":"2024-09-13T05:19:37.137738+0000","last_fullsized":"2024-09-13T05:19:37.137738+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T05:49:15.311928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092053199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"57'40","reported_seq":187,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.289965+0000","last_change":"2024-09-13T05:19:35.195473+0000","last_active":"2024-09-13T05:19:37.289965+0000","last_peered":"2024-09-13T05:19:37.289965+0000","last_clean":"2024-09-13T05:19:37.289965+0000","last_became_active":"2024-09-13T05:19:35.077291+0000","last_became_peered":"2024-09-13T05:19:35.077291+0000","last_unstale":"2024-09-13T05:19:37.289965+0000","last_undegraded":"2024-09-13T05:19:37.289965+0000","last_fullsized":"2024-09-13T05:19:37.289965+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:50.348812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":16384,"num_write":40,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137908+0000","last_change":"2024-09-13T05:19:35.076046+0000","last_active":"2024-09-13T05:19:37.137908+0000","last_peered":"2024-09-13T05:19:37.137908+0000","last_clean":"2024-09-13T05:19:37.137908+0000","last_became_active":"2024-09-13T05:19:35.074375+0000","last_became_peered":"2024-09-13T05:19:35.074375+0000","last_unstale":"2024-09-13T05:19:37.137908+0000","last_undegraded":"2024-09-13T05:19:37.137908+0000","last_fullsized":"2024-09-13T05:19:37.137908+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:58.769823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089178300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"57'28","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.510695+0000","last_change":"2024-09-13T05:19:35.179253+0000","last_active":"2024-09-13T05:19:37.510695+0000","last_peered":"2024-09-13T05:19:37.510695+0000","last_clean":"2024-09-13T05:19:37.510695+0000","last_became_active":"2024-09-13T05:19:35.076916+0000","last_became_peered":"2024-09-13T05:19:35.076916+0000","last_unstale":"2024-09-13T05:19:37.510695+0000","last_undegraded":"2024-09-13T05:19:37.510695+0000","last_fullsized":"2024-09-13T05:19:37.510695+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:40:25.275387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":12315,"num_write":26,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.137789+0000","last_change":"2024-09-13T05:19:35.078422+0000","last_active":"2024-09-13T05:19:37.137789+0000","last_peered":"2024-09-13T05:19:37.137789+0000","last_clean":"2024-09-13T05:19:37.137789+0000","last_became_active":"2024-09-13T05:19:35.076279+0000","last_became_peered":"2024-09-13T05:19:35.076279+0000","last_unstale":"2024-09-13T05:19:37.137789+0000","last_undegraded":"2024-09-13T05:19:37.137789+0000","last_fullsized":"2024-09-13T05:19:37.137789+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:47.540962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012842699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"57'80","reported_seq":448,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T05:19:37.783130+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:19:37.783130+0000","last_peered":"2024-09-13T05:19:37.783130+0000","last_clean":"2024-09-13T05:19:37.783130+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:19:37.783130+0000","last_undegraded":"2024-09-13T05:19:37.783130+0000","last_fullsized":"2024-09-13T05:19:37.783130+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":16777231,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":342,"num_read_kb":32620,"num_write":80,"num_write_kb":40961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":130023550,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2305,"num_read_kb":160246,"num_write":524,"num_write_kb":258224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":255893504,"data_stored":255852796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":487,"ondisk_log_size":487,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607575,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71244,"kb_used_data":43332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94300596,"statfs":{"total":96636764160,"available":96563810304,"internally_reserved":0,"allocated":44371968,"data_stored":43954863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.72299999999999998}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.70199999999999996}]},{"osd":2,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.66100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]}]},{"osd":4,"up_from":54,"seq":231928233986,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69704,"kb_used_data":41592,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94302136,"statfs":{"total":96636764160,"available":96565387264,"internally_reserved":0,"allocated":42590208,"data_stored":42184793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]}]},{"osd":3,"up_from":29,"seq":124554051596,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41020,"kb_used_data":12908,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94330820,"statfs":{"total":96636764160,"available":96594759680,"internally_reserved":0,"allocated":13217792,"data_stored":12814726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607574,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72712,"kb_used_data":37504,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94299128,"statfs":{"total":96636764160,"available":96562307072,"internally_reserved":0,"allocated":38404096,"data_stored":37990546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.51200000000000001}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.625},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.55400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":5,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.871},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.78200000000000003}]}]},{"osd":1,"up_from":12,"seq":51539607574,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101968,"kb_used_data":66760,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94269872,"statfs":{"total":96636764160,"available":96532348928,"internally_reserved":0,"allocated":68362240,"data_stored":67941057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.41999999999999998}]},{"osd":2,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":5,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.625}]}]},{"osd":0,"up_from":12,"seq":51539607574,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61460,"kb_used_data":30336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94310380,"statfs":{"total":96636764160,"available":96573829120,"internally_reserved":0,"allocated":31064064,"data_stored":30650491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":5,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.86199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46145536,"data_stored":46137378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67121152,"data_stored":67108936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37752832,"data_stored":37748774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12591104,"data_stored":12582966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41943040,"data_stored":41943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50331702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:19:40.870 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T05:19:40.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:19:40.871 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-13T05:19:40.871 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T05:19:40.871 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T05:19:40.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-13T05:19:40.910 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:40.910+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.9172513750000002, 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:19:40.965 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:19:40.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:19:40.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8808507969999999, 2024-09-13T05:19:40.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:19:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.974 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8772674929999997, 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:19:40.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:19:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:19:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:19:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:19:40.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.979 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8726334070000004, 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:19:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:19:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:19:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:19:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:19:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:19:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.985 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8693961310000002, 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.986 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:19:40.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:19:40.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8657167320000001, 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:19:40.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.862360539, 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:19:40.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8560292460000003, 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:40.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:40.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.001 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8483492970000004, 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:19:41.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:19:41.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.845718153, 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:19:41.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:19:41.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:19:41.010 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.010 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.010 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.010 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:41.010 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:41.010 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8418761930000001, 2024-09-13T05:19:41.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:19:41.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:41.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:19:41.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:41.015 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8380407969999997, 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:41.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:19:41.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:19:41.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:41.020 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.834585551, 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:41.024 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8305117109999998, 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:41.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:41.028 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8274327819999998, 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8231897660000005, 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:41.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8193137640000003, 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.037 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:19:41.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:41.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:19:41.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:19:41.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:19:41.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:19:41.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8116881390000001, 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:19:41.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:19:41.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:19:41.049 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.049 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.049 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.049 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:41.049 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.808421729, 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:41.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8047293260000004, 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:41.054 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:41.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:41.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:41.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:41.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:41.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:19:41.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:41.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:19:41.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:19:41.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:41.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:19:41.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:19:41.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:41.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:41.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:41.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:19:41.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:19:41.059 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.059 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.059 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.059 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:41.059 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:41.059 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:41.062 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:41.062 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T05:19:41.063 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:41.063 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:41.058+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:41.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:41.064 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:41.064 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd snap ls rbd/testrbdnbd21435 2024-09-13T05:19:41.064 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-13T05:19:41.064 INFO:tasks.workunit.client.0.smithi122.stderr:+ SNAPID=4 2024-09-13T05:19:41.064 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd21435 2024-09-13T05:19:41.065 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:41.065 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:41.065 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:41.065 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:41.065 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:41.065 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:41.065 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd21435 2024-09-13T05:19:41.162 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:41.162+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:41.191 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:41.191 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:41.191 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:41.191 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:41.191 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:41.192 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:41.237 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:41.237 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:41.237 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:41.237 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:41.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:41.256 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=22767 2024-09-13T05:19:41.256 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 22767 2024-09-13T05:19:41.256 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 22767 -C rbd-nbd 2024-09-13T05:19:41.263 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:41.257+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:41.264 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:41.264 INFO:tasks.workunit.client.0.smithi122.stdout: 22767 ? 00:00:00 rbd-nbd 2024-09-13T05:19:41.264 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device '--snap-id 4 testrbdnbd21435' 22767 2024-09-13T05:19:41.265 INFO:tasks.workunit.client.0.smithi122.stderr:+ local 'args=--snap-id 4 testrbdnbd21435' 2024-09-13T05:19:41.265 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=22767 2024-09-13T05:19:41.265 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 4 testrbdnbd21435 2024-09-13T05:19:41.265 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:41.265 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:41.266 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:41.266 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:41.268 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:41.269 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:41.269 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 4 testrbdnbd21435 2024-09-13T05:19:41.361 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:41.361 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^22767\b' 2024-09-13T05:19:41.361 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^22767\b' 2024-09-13T05:19:41.364 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:41.361+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:41.411 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:41.412 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:41.412 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *22767\b' 2024-09-13T05:19:41.412 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *22767\b' 2024-09-13T05:19:41.419 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:41.420 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:41.420 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:41.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:41.465 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:41.461+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:41.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:41.560 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 84.425872361000003, 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:19:41.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:19:41.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.416090831000005, 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:41.568 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:41.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:19:41.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.569 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:41.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:19:41.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.193260578, 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.186230983000002, 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:19:41.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:19:41.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:19:41.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:19:41.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.145448148, 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:41.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:41.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:19:41.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.138589674999999, 2024-09-13T05:19:41.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:19:41.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:19:41.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:19:41.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:64 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [watch unwatch cookie 139805748911888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.116752539, 2024-09-13T05:19:41.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012844899999999999, 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 64 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442441+0000", 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442448+0000", 2024-09-13T05:19:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6540000000000002e-06 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442465+0000", 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7164999999999999e-05 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442472+0000", 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2599999999999999e-06 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442630+0000", 2024-09-13T05:19:41.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015837800000000001 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442736+0000", 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000105449 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443603+0000", 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00086757000000000002 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443639+0000", 2024-09-13T05:19:41.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5806999999999998e-05 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443727+0000", 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7646999999999996e-05 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:19:41.601 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.106447714000002, 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:41.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:19:41.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:41.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:19:41.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:19:41.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:19:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:19:41.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:19:41.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.610 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.610 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:98 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:41.610 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.218689757, 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001341128, 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 98 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340508+0000", 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1829999999999998e-06 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340515+0000", 2024-09-13T05:19:41.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4279999999999999e-06 2024-09-13T05:19:41.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340520+0000", 2024-09-13T05:19:41.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9309999999999999e-06 2024-09-13T05:19:41.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341329+0000", 2024-09-13T05:19:41.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080829399999999998 2024-09-13T05:19:41.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341460+0000", 2024-09-13T05:19:41.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131271 2024-09-13T05:19:41.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341846+0000", 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038646500000000001 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:101 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [sparse-read 659456~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.342590+0000", 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.216604909999999, 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012068000000000001, 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 101 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342590+0000", 2024-09-13T05:19:41.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342590+0000", 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342589+0000", 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342596+0000", 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1640000000000004e-06 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342616+0000", 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9879000000000001e-05 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342631+0000", 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4922999999999999e-05 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.342996+0000", 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000365125 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.343185+0000", 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000189495 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.343797+0000", 2024-09-13T05:19:41.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000611324 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:230 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.072624811000001, 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001614084, 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:41.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 230 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486575+0000", 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3340000000000001e-06 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486581+0000", 2024-09-13T05:19:41.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4689999999999996e-06 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486584+0000", 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8010000000000001e-06 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.487120+0000", 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053554000000000004 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.487394+0000", 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027411100000000001 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.488184+0000", 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079022900000000004 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:233 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 659456~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:41.625 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.069963745000001, 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014141659999999999, 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 233 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489234+0000", 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2660000000000002e-06 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489241+0000", 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6070000000000002e-06 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489246+0000", 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.656e-06 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489607+0000", 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000360711 2024-09-13T05:19:41.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489893+0000", 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028591100000000002 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.490645+0000", 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075255999999999999 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:236 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1052672~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.067484644, 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013904989999999999, 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.630 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491709+0000", 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491716+0000", 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491733+0000", 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7652999999999999e-05 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491742+0000", 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1800000000000002e-06 2024-09-13T05:19:41.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.492084+0000", 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341479 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.492348+0000", 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026391400000000002 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.493101+0000", 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075303000000000002 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:238 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.061864176, 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015224399999999999, 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 238 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497330+0000", 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497333+0000", 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2030000000000001e-06 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497341+0000", 2024-09-13T05:19:41.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4200000000000001e-06 2024-09-13T05:19:41.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497345+0000", 2024-09-13T05:19:41.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1679999999999997e-06 2024-09-13T05:19:41.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497879+0000", 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053369599999999995 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.498107+0000", 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000228435 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.498853+0000", 2024-09-13T05:19:41.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074610300000000002 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.051292427, 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012715370000000001, 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507905+0000", 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9409999999999999e-06 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507914+0000", 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8729999999999996e-06 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507918+0000", 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9009999999999996e-06 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.508538+0000", 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00062024300000000003 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.508662+0000", 2024-09-13T05:19:41.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012431000000000001 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.509174+0000", 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00051175600000000004 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:261 2.2 2:59ccbc16:::rbd_data.11d2ef348a4.0000000000000006:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.037550146999999, 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001208492, 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:41.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521644+0000", 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521646+0000", 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8950000000000001e-06 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521671+0000", 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4397999999999999e-05 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521675+0000", 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4329999999999997e-06 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522277+0000", 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060205000000000002 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522382+0000", 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104731 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522853+0000", 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047135200000000002 2024-09-13T05:19:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:287 2.2 2:59ccbc16:::rbd_data.11d2ef348a4.0000000000000006:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.002334770999999, 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014822030000000001, 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.649 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 287 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:41.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556859+0000", 2024-09-13T05:19:41.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:41.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556864+0000", 2024-09-13T05:19:41.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:19:41.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556878+0000", 2024-09-13T05:19:41.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3457e-05 2024-09-13T05:19:41.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556885+0000", 2024-09-13T05:19:41.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9060000000000002e-06 2024-09-13T05:19:41.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.557685+0000", 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079915800000000001 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.557810+0000", 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000125084 2024-09-13T05:19:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.558342+0000", 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053261499999999995 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4924862399999999, 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:19:41.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:19:41.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:19:41.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4923250140000004, 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:19:41.660 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:19:41.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:19:41.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:19:41.664 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4883.0:10 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:39.429412+0000", 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.1297829429999999, 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897970000000001, 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4883", 2024-09-13T05:19:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 10 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429412+0000", 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429412+0000", 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429410+0000", 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429416+0000", 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4090000000000002e-06 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429424+0000", 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4959999999999996e-06 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:41.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429432+0000", 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5809999999999999e-06 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429496+0000", 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3948999999999994e-05 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429504+0000", 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2339999999999994e-06 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.431013+0000", 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015087869999999999 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.431094+0000", 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0631999999999996e-05 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.431202+0000", 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010819699999999999 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:41.670 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:41.670 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:41.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:41.672 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:41.672 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:41.672 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:41.673 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:41.674 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:41.674 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:41.674 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:41.670+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:41.691 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:41.691 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:41.711 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:41.712 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:41.774 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:41.774+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:41.811 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:41.811 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:41.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:41.872 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:41.873+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:41.921 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:41.921 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/ns/testrbdnbd21435@snap 2024-09-13T05:19:41.973 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:41.970+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:41.996 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:41.996 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:42.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:42.048 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T05:19:42.069 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd21435@snap 2024-09-13T05:19:42.069 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:42.070 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:42.070 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:42.070 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:42.070 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:42.070 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:42.070 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd21435@snap 2024-09-13T05:19:42.074 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:42.069+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:42.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:19:42.175 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:42.173+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:42.182 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:42.182 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:42.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:42.203 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:42.203 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-09-13T05:19:42.203 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:42.203 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-09-13T05:19:42.204 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:42.204 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:42.273 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=22864 2024-09-13T05:19:42.273 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 22864 2024-09-13T05:19:42.273 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 22864 -C rbd-nbd 2024-09-13T05:19:42.276 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:42.274+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:42.276 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:42.276 INFO:tasks.workunit.client.0.smithi122.stdout: 22864 ? 00:00:00 rbd-nbd 2024-09-13T05:19:42.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device rbd/ns/testrbdnbd21435@snap 22864 2024-09-13T05:19:42.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=rbd/ns/testrbdnbd21435@snap 2024-09-13T05:19:42.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=22864 2024-09-13T05:19:42.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd21435@snap 2024-09-13T05:19:42.278 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:42.278 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:42.278 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:42.279 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:42.279 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:42.279 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:42.279 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd21435@snap 2024-09-13T05:19:42.311 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:42.312 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:42.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:42.351 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:42.351 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^22864\b' 2024-09-13T05:19:42.351 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^22864\b' 2024-09-13T05:19:42.401 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:42.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:42.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *22864\b' 2024-09-13T05:19:42.402 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *22864\b' 2024-09-13T05:19:42.408 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:42.408 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:42.414 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:42.414+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:42.478 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:42.478+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:42.494 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:42.494 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:42.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:42.594 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:42.594+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:42.676 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:42.677 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:42.679 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:42.678+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:42.691 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:42.780 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:42.778+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:42.821 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:42.822 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:42.835 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:42.881 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:42.877+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:42.909 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:42.910 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd snap ls rbd/ns/testrbdnbd21435 2024-09-13T05:19:42.910 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-13T05:19:42.960 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:42.960 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:42.971 INFO:tasks.workunit.client.0.smithi122.stderr:+ SNAPID=5 2024-09-13T05:19:42.972 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd21435 2024-09-13T05:19:42.972 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:42.972 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:42.973 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:42.973 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:42.974 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:42.974 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:42.974 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd21435 2024-09-13T05:19:42.975 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:43.019 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:43.017+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:43.083 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:43.082+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:43.115 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:43.115 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-09-13T05:19:43.115 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:43.115 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-09-13T05:19:43.116 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:43.116 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:43.124 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:19:43.147 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:43.147 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:43.161 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:43.180 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=22948 2024-09-13T05:19:43.180 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 22948 2024-09-13T05:19:43.180 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 22948 -C rbd-nbd 2024-09-13T05:19:43.186 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:43.186 INFO:tasks.workunit.client.0.smithi122.stdout: 22948 ? 00:00:00 rbd-nbd 2024-09-13T05:19:43.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device '--snap-id 5 rbd/ns/testrbdnbd21435' 22948 2024-09-13T05:19:43.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ local 'args=--snap-id 5 rbd/ns/testrbdnbd21435' 2024-09-13T05:19:43.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=22948 2024-09-13T05:19:43.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd21435 2024-09-13T05:19:43.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:43.187 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:43.188 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:43.188 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:43.189 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:43.189 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:43.189 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd21435 2024-09-13T05:19:43.194 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:43.194+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:43.280 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:43.280 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^22948\b' 2024-09-13T05:19:43.280 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^22948\b' 2024-09-13T05:19:43.284 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:43.284 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:43.284 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:43.281+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:43.298 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:43.334 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:43.335 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:43.335 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *22948\b' 2024-09-13T05:19:43.335 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *22948\b' 2024-09-13T05:19:43.341 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:43.341 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:43.385 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:43.382+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:43.468 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:43.468 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:43.482 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:43.485 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:43.481+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:43.586 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:43.586+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:43.602 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:43.602 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:43.617 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:43.687 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:43.685+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:43.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:43.739 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:43.740 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:43.754 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:43.795 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:43.793+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:43.842 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:43.844 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd21435 2024-09-13T05:19:43.844 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:43.844 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:43.845 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:43.845 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:43.848 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:43.848 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:43.848 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd21435 2024-09-13T05:19:43.889 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:43.889 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:43.889 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:43.885+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:43.905 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:43.905 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:43.917 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:43.917 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:43.921 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:43.931 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:43.940 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:43.960 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:43.961 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:43.979 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:43.987 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:43.988 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-09-13T05:19:43.988 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:43.988 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-09-13T05:19:43.988 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:43.988 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:43.989 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:43.990+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:44.051 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23010 2024-09-13T05:19:44.051 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23010 2024-09-13T05:19:44.051 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23010 -C rbd-nbd 2024-09-13T05:19:44.057 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:44.057 INFO:tasks.workunit.client.0.smithi122.stdout: 23010 ? 00:00:00 rbd-nbd 2024-09-13T05:19:44.058 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd21435' 23010 2024-09-13T05:19:44.058 INFO:tasks.workunit.client.0.smithi122.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd21435' 2024-09-13T05:19:44.058 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=23010 2024-09-13T05:19:44.058 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd21435 2024-09-13T05:19:44.058 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:44.059 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:44.059 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:44.059 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:44.060 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:44.060 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:44.060 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd21435 2024-09-13T05:19:44.091 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:44.090+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:44.101 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.0540211409999998, 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:19:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.0176205629999995, 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:19:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.0140372590000002, 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:19:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:19:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.0094031730000008, 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.0061658970000007, 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:19:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.0024864979999997, 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9991303049999996, 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.131 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9927990120000008, 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:44.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:19:44.135 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9851190629999991, 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.136 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:19:44.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:19:44.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9824879190000004, 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.141 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9786459589999996, 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.148 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9748105630000001, 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:44.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:19:44.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:19:44.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9713553170000004, 2024-09-13T05:19:44.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:19:44.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:19:44.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9672814770000002, 2024-09-13T05:19:44.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:19:44.158 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:19:44.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9642025479999994, 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:19:44.160 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:19:44.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:44.164 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9599595319999992, 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:19:44.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:44.168 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9560835300000008, 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:44.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9484579049999997, 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:19:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:19:44.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.176 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9451914949999995, 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:19:44.180 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9414990920000008, 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:44.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:44.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:19:44.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:19:44.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:44.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:19:44.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:19:44.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:44.185 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:44.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:44.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23010\b' 2024-09-13T05:19:44.187 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23010\b' 2024-09-13T05:19:44.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:44.191 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:44.189+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:44.197 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:44.198 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:44.198 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *23010\b' 2024-09-13T05:19:44.198 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *23010\b' 2024-09-13T05:19:44.204 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:44.204 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:44.292 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:44.293+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:44.322 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:44.322 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:44.323 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:44.323 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:44.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:44.392 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:44.389+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:44.457 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:44.457 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:44.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:44.493 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:44.493+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:44.594 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:44.593+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.467006067, 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:19:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 86.457224537000002, 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:19:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:19:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.234394284, 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:19:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:19:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.227364689000002, 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:19:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:19:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:19:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.186581854, 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:19:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:19:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.179723380999999, 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:19:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:19:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:19:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:64 2.6 2:63958958:ns::rbd_header.11db3aaeebba:head [watch unwatch cookie 139805748911888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.157886245, 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012844899999999999, 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 64 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442442+0000", 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442441+0000", 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442448+0000", 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6540000000000002e-06 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442465+0000", 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7164999999999999e-05 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442472+0000", 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2599999999999999e-06 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442630+0000", 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015837800000000001 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.442736+0000", 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000105449 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443603+0000", 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00086757000000000002 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443639+0000", 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5806999999999998e-05 2024-09-13T05:19:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.443727+0000", 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7646999999999996e-05 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.147581420000002, 2024-09-13T05:19:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:19:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:19:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:19:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:19:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:98 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.259823463, 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001341128, 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 98 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340505+0000", 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340508+0000", 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1829999999999998e-06 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340515+0000", 2024-09-13T05:19:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4279999999999999e-06 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.340520+0000", 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9309999999999999e-06 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341329+0000", 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080829399999999998 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341460+0000", 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131271 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.341846+0000", 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038646500000000001 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:230 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.113758517000001, 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001614084, 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 230 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486570+0000", 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486575+0000", 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3340000000000001e-06 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486581+0000", 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4689999999999996e-06 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.486584+0000", 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8010000000000001e-06 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.487120+0000", 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053554000000000004 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.487394+0000", 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027411100000000001 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.488184+0000", 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079022900000000004 2024-09-13T05:19:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:233 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 659456~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.111097450999999, 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014141659999999999, 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 233 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489231+0000", 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489234+0000", 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2660000000000002e-06 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489241+0000", 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6070000000000002e-06 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489246+0000", 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.656e-06 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489607+0000", 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000360711 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.489893+0000", 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028591100000000002 2024-09-13T05:19:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.490645+0000", 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075255999999999999 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:236 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1052672~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.10861835, 2024-09-13T05:19:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013904989999999999, 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491710+0000", 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491709+0000", 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491716+0000", 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2024-09-13T05:19:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491733+0000", 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7652999999999999e-05 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.491742+0000", 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1800000000000002e-06 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.492084+0000", 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341479 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.492348+0000", 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026391400000000002 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.493101+0000", 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075303000000000002 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:238 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.102997882, 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015224399999999999, 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 238 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497331+0000", 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497330+0000", 2024-09-13T05:19:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497333+0000", 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2030000000000001e-06 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497341+0000", 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4200000000000001e-06 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497345+0000", 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1679999999999997e-06 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.497879+0000", 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053369599999999995 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.498107+0000", 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000228435 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.498853+0000", 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074610300000000002 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.092426133, 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012715370000000001, 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507902+0000", 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507905+0000", 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9409999999999999e-06 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507914+0000", 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8729999999999996e-06 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.507918+0000", 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9009999999999996e-06 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.508538+0000", 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00062024300000000003 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.508662+0000", 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012431000000000001 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.509174+0000", 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00051175600000000004 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:261 2.2 2:59ccbc16:::rbd_data.11d2ef348a4.0000000000000006:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.078683853000001, 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001208492, 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521645+0000", 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521644+0000", 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521646+0000", 2024-09-13T05:19:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8950000000000001e-06 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521671+0000", 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4397999999999999e-05 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.521675+0000", 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4329999999999997e-06 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522277+0000", 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060205000000000002 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522382+0000", 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104731 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.522853+0000", 2024-09-13T05:19:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047135200000000002 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:287 2.2 2:59ccbc16:::rbd_data.11d2ef348a4.0000000000000006:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.043468477000001, 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014822030000000001, 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 287 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556860+0000", 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556859+0000", 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556864+0000", 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:19:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556878+0000", 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3457e-05 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.556885+0000", 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9060000000000002e-06 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.557685+0000", 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079915800000000001 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.557810+0000", 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000125084 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.558342+0000", 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053261499999999995 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.533619946, 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.5334587200000005, 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:19:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:19:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4883.0:10 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-09-13T05:19:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:39.429412+0000", 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.1709166489999996, 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897970000000001, 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4883", 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 10 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429412+0000", 2024-09-13T05:19:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429412+0000", 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429410+0000", 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429416+0000", 2024-09-13T05:19:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4090000000000002e-06 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429424+0000", 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4959999999999996e-06 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429432+0000", 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5809999999999999e-06 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429496+0000", 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3948999999999994e-05 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.429504+0000", 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2339999999999994e-06 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.431013+0000", 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015087869999999999 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.431094+0000", 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0631999999999996e-05 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:39.431202+0000", 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010819699999999999 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5262.0:67 2.6 e62/54 2:63958958:ns::rbd_header.11db3aaeebba:head v 62'40, mlcod=62'38)", 2024-09-13T05:19:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:44.133611+0000", 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.46671751700000003, 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001481572, 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.133611+0000", 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.133611+0000", 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.133609+0000", 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.133626+0000", 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6963e-05 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.133656+0000", 2024-09-13T05:19:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9519999999999999e-05 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.133669+0000", 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3813e-05 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.133787+0000", 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011793799999999999 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.133827+0000", 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9521999999999999e-05 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.134982+0000", 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011551980000000001 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.135093+0000", 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110605 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:44.708 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd21435 --snap snap 2024-09-13T05:19:44.708 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:44.708 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:44.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:44.710 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:44.709+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:44.710 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:44.710 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:44.711 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:44.711 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:44.711 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd21435 --snap snap 2024-09-13T05:19:44.809 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:44.805+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:44.839 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:44.839 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:44.852 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:44.852 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-09-13T05:19:44.852 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:44.852 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-09-13T05:19:44.852 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:44.852 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:44.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:44.909 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:44.906+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:44.921 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23072 2024-09-13T05:19:44.921 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23072 2024-09-13T05:19:44.921 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23072 -C rbd-nbd 2024-09-13T05:19:44.927 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:44.927 INFO:tasks.workunit.client.0.smithi122.stdout: 23072 ? 00:00:00 rbd-nbd 2024-09-13T05:19:44.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd21435 --snap snap' 23072 2024-09-13T05:19:44.928 INFO:tasks.workunit.client.0.smithi122.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd21435 --snap snap' 2024-09-13T05:19:44.928 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=23072 2024-09-13T05:19:44.928 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd21435 --snap snap 2024-09-13T05:19:44.928 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:44.928 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:44.929 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:44.929 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:44.931 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:44.931 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:44.932 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd21435 --snap snap 2024-09-13T05:19:44.978 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:44.979 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:44.993 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:45.010 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:45.005+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:45.016 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:45.016 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23072\b' 2024-09-13T05:19:45.016 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23072\b' 2024-09-13T05:19:45.063 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:45.064 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:45.064 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *23072\b' 2024-09-13T05:19:45.064 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *23072\b' 2024-09-13T05:19:45.070 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:45.070 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:45.112 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:45.110+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:45.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:19:45.161 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 95.207126846999998, 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012511427, 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952204+0000", 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952369+0000", 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016570500000000001 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952388+0000", 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924e-05 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952398+0000", 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8730000000000002e-06 2024-09-13T05:19:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958423+0000", 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060249700000000002 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958604+0000", 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181075 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.958881+0000", 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00027649400000000001 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961357+0000", 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024759949999999999 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964578+0000", 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032215490000000002 2024-09-13T05:19:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964658+0000", 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9981999999999994e-05 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.964716+0000", 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7460000000000002e-05 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 95.206919689000003, 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012822544999999999, 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 88 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952411+0000", 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952417+0000", 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5690000000000001e-06 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952427+0000", 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5000000000000005e-06 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.952434+0000", 2024-09-13T05:19:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5939999999999996e-06 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.959927+0000", 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074927420000000002 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960057+0000", 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013036499999999999 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.960286+0000", 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022870599999999999 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.961780+0000", 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014937150000000001 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965111+0000", 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033308840000000001 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965182+0000", 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1328999999999994e-05 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:09.965234+0000", 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1966999999999999e-05 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 95.028410449000006, 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011063511, 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 117 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130921+0000", 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130920+0000", 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130923+0000", 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130930+0000", 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1620000000000002e-06 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.130933+0000", 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0089999999999999e-06 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136419+0000", 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054856549999999999 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136485+0000", 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6347999999999996e-05 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.136570+0000", 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4078000000000004e-05 2024-09-13T05:19:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.137540+0000", 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097037700000000002 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141929+0000", 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043893910000000003 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141961+0000", 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1661000000000001e-05 2024-09-13T05:19:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.141984+0000", 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3264e-05 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.644030126000004, 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012724638, 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 132 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515301+0000", 2024-09-13T05:19:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515300+0000", 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515307+0000", 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9600000000000003e-06 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515317+0000", 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8630000000000003e-06 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515324+0000", 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0929999999999997e-06 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519325+0000", 2024-09-13T05:19:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040009090000000004 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519484+0000", 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015855200000000001 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.519767+0000", 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028310500000000002 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523649+0000", 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038823600000000001 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527904+0000", 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042551710000000003 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527975+0000", 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0789000000000003e-05 2024-09-13T05:19:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528026+0000", 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0745999999999998e-05 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.643992990000001, 2024-09-13T05:19:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012945353999999999, 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 133 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515338+0000", 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515343+0000", 2024-09-13T05:19:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6930000000000004e-06 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515353+0000", 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3999999999999998e-06 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515359+0000", 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8299999999999998e-06 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520492+0000", 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051324930000000001 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520620+0000", 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012776899999999999 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.520814+0000", 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019421900000000001 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525051+0000", 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042366909999999999 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528155+0000", 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031044190000000002 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528224+0000", 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9406999999999995e-05 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528284+0000", 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9116000000000001e-05 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.643887070999995, 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.643615045999994, 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.643571460000004, 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.643539232999998, 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:45.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.643328084999993, 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.643291791999999, 2024-09-13T05:19:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.643253341999994, 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.643078439999996, 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:45.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.236 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.828374501999999, 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:45.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:19:45.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:19:45.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:19:45.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.820844389000001, 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:19:45.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:19:45.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.811275062, 2024-09-13T05:19:45.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:45.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.252 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.80011219, 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:45.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:45.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:19:45.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.793952593, 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:19:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:19:45.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:19:45.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.745116812999999, 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:19:45.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:19:45.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:19:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.736840278999999, 2024-09-13T05:19:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:19:45.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:19:45.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.274 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.274 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.274 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:45.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:45.276 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:45.274+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:45.376 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:45.377+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:45.434 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:45.435 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:45.435 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:45.435 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:45.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:45.494 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:45.494+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:45.565 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:45.565 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:45.565 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:45.565 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:45.565 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:45.571 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:45.572 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:45.572 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:45.572 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:45.574 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:45.575 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:45.577 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:45.577 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:45.577 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:45.578 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:45.577+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:45.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:45.678 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:45.678+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:45.719 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:45.720 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:45.720 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:45.720 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:45.720 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:45.720 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:45.754 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:45.754 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:45.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:45.754 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:45.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.423973992000001, 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:19:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:19:45.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.412768231000001, 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:45.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.402589176999999, 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:19:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:19:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:19:45.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.395153374, 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:19:45.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:19:45.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.388545670999999, 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:19:45.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:19:45.781 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.379319453000001, 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:19:45.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.785 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4797.0:1 3.3 e48/47 3:c4c92e5a:::rbd_trash:head v 48'1, mlcod=0'0)", 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.291825539000001, 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015947019999999999, 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459810+0000", 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8649999999999998e-06 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459820+0000", 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9899999999999992e-06 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459829+0000", 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.431e-06 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459885+0000", 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5868000000000001e-05 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459912+0000", 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7203e-05 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461364+0000", 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014515820000000001 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461399+0000", 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5063e-05 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4664.0:119 2.3 e50/16 2:d073633c:::rbd_header.11d2ef348a4:head v 50'39, mlcod=50'37)", 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:26.364298+0000", 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.387332226000002, 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013753039999999999, 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364298+0000", 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364298+0000", 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364296+0000", 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364309+0000", 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3246e-05 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364340+0000", 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0216e-05 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364352+0000", 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2327000000000001e-05 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364454+0000", 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010179399999999999 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.364497+0000", 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2916e-05 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.365601+0000", 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001104397 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:26.365673+0000", 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2047999999999999e-05 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:180 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.327279407999999, 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013961119999999999, 2024-09-13T05:19:45.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 180 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424350+0000", 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424354+0000", 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3560000000000003e-06 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424365+0000", 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0832999999999999e-05 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424374+0000", 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5599999999999994e-06 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424997+0000", 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00062386799999999997 2024-09-13T05:19:45.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.425131+0000", 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133614 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.425747+0000", 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00061550899999999998 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:494 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.856467+0000", 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.895162844000001, 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001401575, 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 494 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856467+0000", 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856467+0000", 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856466+0000", 2024-09-13T05:19:45.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856473+0000", 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4839999999999998e-06 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856493+0000", 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9819000000000001e-05 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856509+0000", 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5333e-05 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.857232+0000", 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00072378299999999998 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.857354+0000", 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121625 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.857869+0000", 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00051458799999999996 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.884610052999999, 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:518 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.888353+0000", 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.863276655, 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013167999999999999, 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 518 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888353+0000", 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888353+0000", 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888353+0000", 2024-09-13T05:19:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888355+0000", 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3819999999999998e-06 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888360+0000", 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0849999999999996e-06 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888364+0000", 2024-09-13T05:19:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8519999999999997e-06 2024-09-13T05:19:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.888978+0000", 2024-09-13T05:19:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00061398300000000002 2024-09-13T05:19:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.889137+0000", 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015835299999999999 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.889670+0000", 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053354100000000001 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:527 2.0 2:0158b2c3:::rbd_data.11d2ef348a4.000000000000000e:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.851949564999998, 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001391878, 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:45.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 527 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899682+0000", 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.198e-06 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899687+0000", 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8760000000000004e-06 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899691+0000", 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6509999999999999e-06 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.900302+0000", 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00061118999999999995 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.900505+0000", 2024-09-13T05:19:45.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000203523 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.901072+0000", 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00056678000000000002 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:551 2.0 2:0158b2c3:::rbd_data.11d2ef348a4.000000000000000e:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.931321+0000", 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.820308806, 2024-09-13T05:19:45.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001380287, 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 551 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931321+0000", 2024-09-13T05:19:45.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931321+0000", 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931321+0000", 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931323+0000", 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.633e-06 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931328+0000", 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0830000000000003e-06 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931332+0000", 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7529999999999999e-06 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.931903+0000", 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000571296 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.932067+0000", 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016321 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.932701+0000", 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00063475200000000004 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4927.0:12 2.3 e56/16 2:d073633c:::rbd_header.11d2ef348a4:head v 56'122, mlcod=55'120)", 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.992223+0000", 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.7594070510000002, 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014957080000000001, 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.823 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992223+0000", 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992223+0000", 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992221+0000", 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992237+0000", 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6656000000000001e-05 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992270+0000", 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2863e-05 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992285+0000", 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5026e-05 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992393+0000", 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010806099999999999 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992458+0000", 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4258999999999996e-05 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.993654+0000", 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011958209999999999 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.993719+0000", 2024-09-13T05:19:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5055000000000006e-05 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4883.0:22 2.3 e60/16 2:d073633c:::rbd_header.11d2ef348a4:head v 60'166, mlcod=59'164)", 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:40.106376+0000", 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.6452537750000005, 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015055159999999999, 2024-09-13T05:19:45.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106376+0000", 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106376+0000", 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106376+0000", 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106390+0000", 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4263999999999999e-05 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106471+0000", 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1422999999999996e-05 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106485+0000", 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4081e-05 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106670+0000", 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018492899999999999 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106731+0000", 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0625000000000003e-05 2024-09-13T05:19:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.107839+0000", 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001108277 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.107882+0000", 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2443e-05 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4889.0:68 2.3 e60/16 2:d073633c:::rbd_header.11d2ef348a4:head v 60'172, mlcod=60'170)", 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:40.432192+0000", 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.3194377490000004, 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001408795, 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432192+0000", 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432192+0000", 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432190+0000", 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432207+0000", 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6393e-05 2024-09-13T05:19:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432238+0000", 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0923000000000002e-05 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432252+0000", 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4032999999999999e-05 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432376+0000", 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012397800000000001 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432431+0000", 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5798999999999999e-05 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.433538+0000", 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00110614 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.433601+0000", 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3441999999999993e-05 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.837 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:48 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 69632~61440] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999956+0000", 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.751673917, 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00139817, 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:19:45.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999956+0000", 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999956+0000", 2024-09-13T05:19:45.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999954+0000", 2024-09-13T05:19:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999958+0000", 2024-09-13T05:19:45.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4190000000000004e-06 2024-09-13T05:19:45.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999965+0000", 2024-09-13T05:19:45.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5529999999999998e-06 2024-09-13T05:19:45.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999979+0000", 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3784000000000001e-05 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001059+0000", 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010799990000000001 2024-09-13T05:19:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001176+0000", 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116995 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001354+0000", 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017862999999999999 2024-09-13T05:19:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.751647194, 2024-09-13T05:19:45.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:19:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:19:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:19:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:19:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.845 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:45.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:45.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:19:45.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:45.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:19:45.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:19:45.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:19:45.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:19:45.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:19:45.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:19:45.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:19:45.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:19:45.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:19:45.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:19:45.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:19:45.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:19:45.851 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.851 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.851 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.851 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4982.0:50 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:5 [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:45.851 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:45.852 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.87665338100000001, 2024-09-13T05:19:45.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001569698, 2024-09-13T05:19:45.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:45.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:45.852 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4982", 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1259763078", 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 50 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874976+0000", 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967296 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874978+0000", 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2100000000000001e-06 2024-09-13T05:19:45.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874981+0000", 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1760000000000002e-06 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874984+0000", 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9569999999999998e-06 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876017+0000", 2024-09-13T05:19:45.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00103304 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876260+0000", 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000243403 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876546+0000", 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028614500000000002 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:45.857 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:45.857 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:45.858 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23134 2024-09-13T05:19:45.858 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23134 2024-09-13T05:19:45.858 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23134 -C rbd-nbd 2024-09-13T05:19:45.859 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:45.859 INFO:tasks.workunit.client.0.smithi122.stdout: 23134 ? 00:00:00 rbd-nbd 2024-09-13T05:19:45.859 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:45.860 INFO:tasks.workunit.client.0.smithi122.stderr:+ pid=23134 2024-09-13T05:19:45.860 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd21435 2024-09-13T05:19:45.860 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:45.860 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:45.861 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:45.861 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:45.861 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:45.861 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:45.861 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd21435 2024-09-13T05:19:45.861 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:45.853+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:45.959 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:45.958+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:45.961 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-09-13T05:19:45.961 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd ns 2024-09-13T05:19:45.961 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:45.962 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns=ns 2024-09-13T05:19:45.962 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:45.962 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd1'\'']/id' 2024-09-13T05:19:45.993 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:45.994 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:45.994 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:45.994 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:45.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:46.008 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:46.034 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23180 2024-09-13T05:19:46.035 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23180 2024-09-13T05:19:46.035 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23180 -C rbd-nbd 2024-09-13T05:19:46.041 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:46.041 INFO:tasks.workunit.client.0.smithi122.stdout: 23134 ? 00:00:00 rbd-nbd 2024-09-13T05:19:46.042 INFO:tasks.workunit.client.0.smithi122.stdout: 23180 ? 00:00:00 rbd-nbd 2024-09-13T05:19:46.042 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device rbd/ns/testrbdnbd21435 23180 2024-09-13T05:19:46.043 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=rbd/ns/testrbdnbd21435 2024-09-13T05:19:46.043 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=23180 2024-09-13T05:19:46.043 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd21435 2024-09-13T05:19:46.043 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:46.043 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:46.043 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:46.044 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:46.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:46.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:46.044 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd21435 2024-09-13T05:19:46.060 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:46.057+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:46.064 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:19:46.064 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:19:46.064 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T05:19:46.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T05:19:46.128 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:46.128 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:46.129 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:46.129 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:46.129 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:46.130 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23180\b' 2024-09-13T05:19:46.130 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:46.130 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23180\b' 2024-09-13T05:19:46.130 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:19:46.143 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:46.155 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:46.156 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:46.160 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:46.158+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:46.172 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:46.188 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:46.188 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:46.190 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:46.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:46.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *23180\b' 2024-09-13T05:19:46.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *23180\b' 2024-09-13T05:19:46.197 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:46.197 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:46.214 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:46.214 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:46.245 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:46.246 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:46.262 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:46.257+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:46.279 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:46.319 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:46.319 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:46.333 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:46.363 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:46.362+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 63, 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5", 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2024-09-13T05:17:57.363809+0000", 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2024-09-13T05:19:46.122528+0000", 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2024-09-13T05:19:34.047702+0000", 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2024-09-13T05:17:59.738330+0000", 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5799936, 2024-09-13T05:19:46.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 7, 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 3, 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2024-09-13T05:19:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "reef", 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_crimson": false, 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": ".mgr", 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:08.122164+0000", 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:19:46.399 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:19:46.400 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "15", 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:19:46.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:19:46.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2024-09-13T05:19:46.403 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": {} 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 6.059999942779541, 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 6.059999942779541, 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.33000001311302185, 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2, 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2, 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:19:46.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:15.001664+0000", 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:19:46.405 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:19:46.406 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:19:46.407 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:19:46.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.407 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "61", 2024-09-13T05:19:46.407 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:19:46.407 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:19:46.407 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:19:46.407 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:19:46.407 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:19:46.407 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 5, 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 61, 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:19:46.408 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:19:46.409 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 2.25, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 2.25, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 1, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2.25, 2024-09-13T05:19:46.410 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2.25, 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 3, 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "new_default_pool21435", 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:19:21.793970+0000", 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:19:46.411 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:19:46.412 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "50", 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:19:46.413 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 3, 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 50, 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:19:46.414 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:19:46.415 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:19:46.416 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 3.0099999904632568, 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 3.0099999904632568, 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.82999998331069946, 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2.5, 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2.5, 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:19:46.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "e58174f4-85ec-4321-8bc3-384261c59c6b", 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 48, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:46.418 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:46.419 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.420 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.421 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.422 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/2153409514", 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/2153409514", 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/2153409514", 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/2153409514", 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:19:46.423 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "84a3db6c-dacd-43d5-b964-37fdef0091a4", 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 54, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/2608485151", 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/2608485151", 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/2608485151", 2024-09-13T05:19:46.428 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/2608485151", 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "0d530476-cf38-4544-9ce7-e8ed35df220b", 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:46.429 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 47, 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:46.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/1006404377", 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/1006404377", 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/1006404377", 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/1006404377", 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:46.434 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "fd91f7e2-580d-47ea-9db5-792c8f1ebe01", 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.980804443359375, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 20, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 29, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 47, 2024-09-13T05:19:46.435 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 21, 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6816", 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6817", 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:46.436 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6818", 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6819", 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:46.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6822", 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6823", 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:46.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6820", 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6821", 2024-09-13T05:19:46.439 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6817/1834661236", 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6819/1834661236", 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6823/1834661236", 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6821/1834661236", 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "01af0f4c-5da2-46fe-a262-59fcc6a2efa9", 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 38, 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 54, 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 54, 2024-09-13T05:19:46.441 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 40, 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6808", 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6809", 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:46.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6810", 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6811", 2024-09-13T05:19:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6814", 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6815", 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6812", 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.445 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6813", 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6809/3457694380", 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6811/3457694380", 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6815/3457694380", 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6813/3457694380", 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "64cdd5f2-1568-4e96-ad90-6ef7414651f0", 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 47, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:46.447 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6800", 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6801", 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6802", 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6803", 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6806", 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6807", 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.450 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6804", 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6805", 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.451 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6801/2778785304", 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6803/2778785304", 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6807/2778785304", 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6805/2778785304", 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2024-09-13T05:19:46.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.193869+0000", 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:46.453 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.255764+0000", 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:46.454 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.286354+0000", 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.221310+0000", 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 21 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.026235+0000", 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 40 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:46.456 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.010242+0000", 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: 0, 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2024-09-13T05:19:46.457 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_primaries": [], 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2024-09-13T05:19:46.458 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2024-09-13T05:19:46.459 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2024-09-13T05:19:46.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:46.460 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:46.461 INFO:tasks.thrashosds.thrasher:j is {'epoch': 63, 'fsid': '1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5', 'created': '2024-09-13T05:17:57.363809+0000', 'modified': '2024-09-13T05:19:46.122528+0000', 'last_up_change': '2024-09-13T05:19:34.047702+0000', 'last_in_change': '2024-09-13T05:17:59.738330+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T05:18:08.122164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T05:18:15.001664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool21435', 'create_time': '2024-09-13T05:19:21.793970+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 50, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e58174f4-85ec-4321-8bc3-384261c59c6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 2153409514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 2153409514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 2153409514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 2153409514}]}, 'public_addr': '172.21.15.16:6801/2153409514', 'cluster_addr': '172.21.15.16:6803/2153409514', 'heartbeat_back_addr': '172.21.15.16:6807/2153409514', 'heartbeat_front_addr': '172.21.15.16:6805/2153409514', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '84a3db6c-dacd-43d5-b964-37fdef0091a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 2608485151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 2608485151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 2608485151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 2608485151}]}, 'public_addr': '172.21.15.16:6809/2608485151', 'cluster_addr': '172.21.15.16:6811/2608485151', 'heartbeat_back_addr': '172.21.15.16:6815/2608485151', 'heartbeat_front_addr': '172.21.15.16:6813/2608485151', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0d530476-cf38-4544-9ce7-e8ed35df220b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 1006404377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 1006404377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 1006404377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 1006404377}]}, 'public_addr': '172.21.15.16:6817/1006404377', 'cluster_addr': '172.21.15.16:6819/1006404377', 'heartbeat_back_addr': '172.21.15.16:6823/1006404377', 'heartbeat_front_addr': '172.21.15.16:6821/1006404377', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fd91f7e2-580d-47ea-9db5-792c8f1ebe01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.980804443359375, 'last_clean_begin': 12, 'last_clean_end': 20, 'up_from': 29, 'up_thru': 47, 'down_at': 21, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6816', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6817', 'nonce': 1834661236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6818', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6819', 'nonce': 1834661236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6822', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6823', 'nonce': 1834661236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6820', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6821', 'nonce': 1834661236}]}, 'public_addr': '172.21.15.79:6817/1834661236', 'cluster_addr': '172.21.15.79:6819/1834661236', 'heartbeat_back_addr': '172.21.15.79:6823/1834661236', 'heartbeat_front_addr': '172.21.15.79:6821/1834661236', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '01af0f4c-5da2-46fe-a262-59fcc6a2efa9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 12, 'last_clean_end': 38, 'up_from': 54, 'up_thru': 54, 'down_at': 40, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6808', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6809', 'nonce': 3457694380}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6810', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6811', 'nonce': 3457694380}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6814', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6815', 'nonce': 3457694380}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6812', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6813', 'nonce': 3457694380}]}, 'public_addr': '172.21.15.79:6809/3457694380', 'cluster_addr': '172.21.15.79:6811/3457694380', 'heartbeat_back_addr': '172.21.15.79:6815/3457694380', 'heartbeat_front_addr': '172.21.15.79:6813/3457694380', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '64cdd5f2-1568-4e96-ad90-6ef7414651f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6800', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6801', 'nonce': 2778785304}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6802', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6803', 'nonce': 2778785304}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6806', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6807', 'nonce': 2778785304}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6804', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6805', 'nonce': 2778785304}]}, 'public_addr': '172.21.15.79:6801/2778785304', 'cluster_addr': '172.21.15.79:6803/2778785304', 'heartbeat_back_addr': '172.21.15.79:6807/2778785304', 'heartbeat_front_addr': '172.21.15.79:6805/2778785304', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.193869+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.255764+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.286354+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.221310+0000', 'dead_epoch': 21}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.026235+0000', 'dead_epoch': 40}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.010242+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [0, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T05:19:46.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:19:46.497 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:46.497 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:46.502 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:46.502+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:46.511 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:46.565 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:46.562+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:46.631 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:46.631 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:46.645 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:46.665 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:46.662+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:46.699 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:46.699 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device rbd/testrbdnbd21435 23134 2024-09-13T05:19:46.699 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=rbd/testrbdnbd21435 2024-09-13T05:19:46.699 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=23134 2024-09-13T05:19:46.699 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap rbd/testrbdnbd21435 2024-09-13T05:19:46.699 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:46.700 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:46.701 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:46.701 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:46.704 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:46.704 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:46.704 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/testrbdnbd21435 2024-09-13T05:19:46.766 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:46.761+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:46.768 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:46.768 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:46.783 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:46.792 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:46.792 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23134\b' 2024-09-13T05:19:46.793 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23134\b' 2024-09-13T05:19:46.798 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:19:46.798 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:19:46.816 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-13T05:19:46.130713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134808317,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":2996,"num_read_kb":164840,"num_write":714,"num_write_kb":264747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":649,"ondisk_log_size":649,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":456308,"kb_used_data":267436,"kb_used_omap":14,"kb_used_meta":188785,"kb_avail":565774732,"statfs":{"total":579820584960,"available":579353325568,"internally_reserved":0,"allocated":273854464,"data_stored":271186108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":193315958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4194380,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":575,"num_read_kb":4494,"num_write":29,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.030115"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":204,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T05:19:40.102712+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:19:40.102712+0000","last_peered":"2024-09-13T05:19:40.102712+0000","last_clean":"2024-09-13T05:19:40.102712+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:19:40.102712+0000","last_undegraded":"2024-09-13T05:19:40.102712+0000","last_fullsized":"2024-09-13T05:19:40.102712+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"60'174","reported_seq":935,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.026051+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:19:42.026051+0000","last_peered":"2024-09-13T05:19:42.026051+0000","last_clean":"2024-09-13T05:19:42.026051+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:19:42.026051+0000","last_undegraded":"2024-09-13T05:19:42.026051+0000","last_fullsized":"2024-09-13T05:19:42.026051+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":12582950,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1625,"num_read_kb":14471,"num_write":189,"num_write_kb":20617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"50'2","reported_seq":26,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-13T05:19:39.096084+0000","last_change":"2024-09-13T05:19:25.480800+0000","last_active":"2024-09-13T05:19:39.096084+0000","last_peered":"2024-09-13T05:19:39.096084+0000","last_clean":"2024-09-13T05:19:39.096084+0000","last_became_active":"2024-09-13T05:19:23.455354+0000","last_became_peered":"2024-09-13T05:19:23.455354+0000","last_unstale":"2024-09-13T05:19:39.096084+0000","last_undegraded":"2024-09-13T05:19:39.096084+0000","last_fullsized":"2024-09-13T05:19:39.096084+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:20.817288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017913759999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"57'58","reported_seq":237,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.221086+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:19:42.221086+0000","last_peered":"2024-09-13T05:19:42.221086+0000","last_clean":"2024-09-13T05:19:42.221086+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:19:42.221086+0000","last_undegraded":"2024-09-13T05:19:42.221086+0000","last_fullsized":"2024-09-13T05:19:42.221086+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":25165824,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":123,"num_read_kb":25100,"num_write":57,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.031605+0000","last_change":"2024-09-13T05:19:35.075887+0000","last_active":"2024-09-13T05:19:42.031605+0000","last_peered":"2024-09-13T05:19:42.031605+0000","last_clean":"2024-09-13T05:19:42.031605+0000","last_became_active":"2024-09-13T05:19:35.073898+0000","last_became_peered":"2024-09-13T05:19:35.073898+0000","last_unstale":"2024-09-13T05:19:42.031605+0000","last_undegraded":"2024-09-13T05:19:42.031605+0000","last_fullsized":"2024-09-13T05:19:42.031605+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:23.211903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010913139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"60'21","reported_seq":116,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T05:19:40.244845+0000","last_change":"2024-09-13T05:18:43.272711+0000","last_active":"2024-09-13T05:19:40.244845+0000","last_peered":"2024-09-13T05:19:40.244845+0000","last_clean":"2024-09-13T05:19:40.244845+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:19:40.244845+0000","last_undegraded":"2024-09-13T05:19:40.244845+0000","last_fullsized":"2024-09-13T05:19:40.244845+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:27.589949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388662,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":43,"num_read_kb":8195,"num_write":21,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":27,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.038439+0000","last_change":"2024-09-13T05:19:25.538085+0000","last_active":"2024-09-13T05:19:42.038439+0000","last_peered":"2024-09-13T05:19:42.038439+0000","last_clean":"2024-09-13T05:19:42.038439+0000","last_became_active":"2024-09-13T05:19:23.462578+0000","last_became_peered":"2024-09-13T05:19:23.462578+0000","last_unstale":"2024-09-13T05:19:42.038439+0000","last_undegraded":"2024-09-13T05:19:42.038439+0000","last_fullsized":"2024-09-13T05:19:42.038439+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:48.193321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056272899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"57'56","reported_seq":307,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:43.065789+0000","last_change":"2024-09-13T05:19:24.517596+0000","last_active":"2024-09-13T05:19:43.065789+0000","last_peered":"2024-09-13T05:19:43.065789+0000","last_clean":"2024-09-13T05:19:43.065789+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:19:43.065789+0000","last_undegraded":"2024-09-13T05:19:43.065789+0000","last_fullsized":"2024-09-13T05:19:43.065789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":171,"num_read_kb":36960,"num_write":56,"num_write_kb":36866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"48'1","reported_seq":28,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.023829+0000","last_change":"2024-09-13T05:19:25.535626+0000","last_active":"2024-09-13T05:19:42.023829+0000","last_peered":"2024-09-13T05:19:42.023829+0000","last_clean":"2024-09-13T05:19:42.023829+0000","last_became_active":"2024-09-13T05:19:23.458938+0000","last_became_peered":"2024-09-13T05:19:23.458938+0000","last_unstale":"2024-09-13T05:19:42.023829+0000","last_undegraded":"2024-09-13T05:19:42.023829+0000","last_fullsized":"2024-09-13T05:19:42.023829+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:28:08.756172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000291315,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.200649+0000","last_change":"2024-09-13T05:19:25.474324+0000","last_active":"2024-09-13T05:19:42.200649+0000","last_peered":"2024-09-13T05:19:42.200649+0000","last_clean":"2024-09-13T05:19:42.200649+0000","last_became_active":"2024-09-13T05:19:23.455224+0000","last_became_peered":"2024-09-13T05:19:23.455224+0000","last_unstale":"2024-09-13T05:19:42.200649+0000","last_undegraded":"2024-09-13T05:19:42.200649+0000","last_fullsized":"2024-09-13T05:19:42.200649+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:43.322756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"61'62","reported_seq":219,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.046663+0000","last_change":"2024-09-13T05:18:43.275214+0000","last_active":"2024-09-13T05:19:42.046663+0000","last_peered":"2024-09-13T05:19:42.046663+0000","last_clean":"2024-09-13T05:19:42.046663+0000","last_became_active":"2024-09-13T05:18:43.274886+0000","last_became_peered":"2024-09-13T05:18:43.274886+0000","last_unstale":"2024-09-13T05:19:42.046663+0000","last_undegraded":"2024-09-13T05:19:42.046663+0000","last_fullsized":"2024-09-13T05:19:42.046663+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:53.129994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":16777254,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":16483,"num_write":78,"num_write_kb":32804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.031595+0000","last_change":"2024-09-13T05:19:35.163055+0000","last_active":"2024-09-13T05:19:42.031595+0000","last_peered":"2024-09-13T05:19:42.031595+0000","last_clean":"2024-09-13T05:19:42.031595+0000","last_became_active":"2024-09-13T05:19:35.161280+0000","last_became_peered":"2024-09-13T05:19:35.161280+0000","last_unstale":"2024-09-13T05:19:42.031595+0000","last_undegraded":"2024-09-13T05:19:42.031595+0000","last_fullsized":"2024-09-13T05:19:42.031595+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T05:49:15.311928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092053199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"57'40","reported_seq":220,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.227138+0000","last_change":"2024-09-13T05:19:35.195473+0000","last_active":"2024-09-13T05:19:42.227138+0000","last_peered":"2024-09-13T05:19:42.227138+0000","last_clean":"2024-09-13T05:19:42.227138+0000","last_became_active":"2024-09-13T05:19:35.077291+0000","last_became_peered":"2024-09-13T05:19:35.077291+0000","last_unstale":"2024-09-13T05:19:42.227138+0000","last_undegraded":"2024-09-13T05:19:42.227138+0000","last_fullsized":"2024-09-13T05:19:42.227138+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:50.348812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":16900,"num_write":40,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.031662+0000","last_change":"2024-09-13T05:19:35.076046+0000","last_active":"2024-09-13T05:19:42.031662+0000","last_peered":"2024-09-13T05:19:42.031662+0000","last_clean":"2024-09-13T05:19:42.031662+0000","last_became_active":"2024-09-13T05:19:35.074375+0000","last_became_peered":"2024-09-13T05:19:35.074375+0000","last_unstale":"2024-09-13T05:19:42.031662+0000","last_undegraded":"2024-09-13T05:19:42.031662+0000","last_fullsized":"2024-09-13T05:19:42.031662+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:58.769823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089178300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"61'37","reported_seq":118,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:43.071103+0000","last_change":"2024-09-13T05:19:35.179253+0000","last_active":"2024-09-13T05:19:43.071103+0000","last_peered":"2024-09-13T05:19:43.071103+0000","last_clean":"2024-09-13T05:19:43.071103+0000","last_became_active":"2024-09-13T05:19:35.076916+0000","last_became_peered":"2024-09-13T05:19:35.076916+0000","last_unstale":"2024-09-13T05:19:43.071103+0000","last_undegraded":"2024-09-13T05:19:43.071103+0000","last_fullsized":"2024-09-13T05:19:43.071103+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:40:25.275387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777254,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":202,"num_read_kb":12433,"num_write":31,"num_write_kb":24582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T05:19:42.031612+0000","last_change":"2024-09-13T05:19:35.078422+0000","last_active":"2024-09-13T05:19:42.031612+0000","last_peered":"2024-09-13T05:19:42.031612+0000","last_clean":"2024-09-13T05:19:42.031612+0000","last_became_active":"2024-09-13T05:19:35.076279+0000","last_became_peered":"2024-09-13T05:19:35.076279+0000","last_unstale":"2024-09-13T05:19:42.031612+0000","last_undegraded":"2024-09-13T05:19:42.031612+0000","last_fullsized":"2024-09-13T05:19:42.031612+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:47.540962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012842699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"58'81","reported_seq":528,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T05:19:41.212092+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:19:41.212092+0000","last_peered":"2024-09-13T05:19:41.212092+0000","last_clean":"2024-09-13T05:19:41.212092+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:19:41.212092+0000","last_undegraded":"2024-09-13T05:19:41.212092+0000","last_fullsized":"2024-09-13T05:19:41.212092+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":16777231,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":423,"num_read_kb":34198,"num_write":81,"num_write_kb":40962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134217930,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2880,"num_read_kb":164740,"num_write":553,"num_write_kb":262329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268492800,"data_stored":268435860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":529,"ondisk_log_size":529,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607576,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77444,"kb_used_data":50424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94294396,"statfs":{"total":96636764160,"available":96557461504,"internally_reserved":0,"allocated":51634176,"data_stored":51184581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.52500000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76700000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]}]},{"osd":4,"up_from":54,"seq":231928233987,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69752,"kb_used_data":41644,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94302088,"statfs":{"total":96636764160,"available":96565338112,"internally_reserved":0,"allocated":42643456,"data_stored":42205551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]}]},{"osd":3,"up_from":29,"seq":124554051597,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53368,"kb_used_data":25260,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94318472,"statfs":{"total":96636764160,"available":96582115328,"internally_reserved":0,"allocated":25866240,"data_stored":25423044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]}]},{"osd":2,"up_from":12,"seq":51539607575,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72764,"kb_used_data":37552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94299076,"statfs":{"total":96636764160,"available":96562253824,"internally_reserved":0,"allocated":38453248,"data_stored":38011266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":5,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.755}]}]},{"osd":1,"up_from":12,"seq":51539607575,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102020,"kb_used_data":66808,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94269820,"statfs":{"total":96636764160,"available":96532295680,"internally_reserved":0,"allocated":68411392,"data_stored":67961777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.66000000000000003}]},{"osd":2,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":5,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.92200000000000004}]}]},{"osd":0,"up_from":12,"seq":51539607575,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80960,"kb_used_data":45748,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94290880,"statfs":{"total":96636764160,"available":96553861120,"internally_reserved":0,"allocated":46845952,"data_stored":46399889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.371},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.63300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46149632,"data_stored":46137416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67121152,"data_stored":67108936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37752832,"data_stored":37748774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:19:46.818 INFO:tasks.thrashosds.thrasher:Setting 3.3 to [1, 0, 4, 5] 2024-09-13T05:19:46.818 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.3', '1', '0', '4', '5'] 2024-09-13T05:19:46.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.3 1 0 4 5 2024-09-13T05:19:46.840 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:46.841 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:46.841 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *23134\b' 2024-09-13T05:19:46.841 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *23134\b' 2024-09-13T05:19:46.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:46.849 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:46.906 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:46.906+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:46.950 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:46.950 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:46.964 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:46.968 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:46.966+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:47.068 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:47.066+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:47.086 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:47.086 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:47.100 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:47.170 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:47.170+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:47.217 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:47.217 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:47.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:47.270 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:47.265+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:47.351 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' rbd = rbd ']' 2024-09-13T05:19:47.351 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd21435 2024-09-13T05:19:47.351 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:47.352 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:47.353 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:47.353 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:47.355 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:47.355 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:47.356 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:47.356 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:47.356 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd21435 2024-09-13T05:19:47.370 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:47.371 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:47.370+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:47.472 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:47.470+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:47.491 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:47.492 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:47.507 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:47.509 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:47.509 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:47.509 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:47.509 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:47.510 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:47.510 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:47.579 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23274 2024-09-13T05:19:47.579 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23274 2024-09-13T05:19:47.579 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23274 -C rbd-nbd 2024-09-13T05:19:47.585 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:47.585 INFO:tasks.workunit.client.0.smithi122.stdout: 23274 ? 00:00:00 rbd-nbd 2024-09-13T05:19:47.586 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device testrbdnbd21435 23274 2024-09-13T05:19:47.586 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=testrbdnbd21435 2024-09-13T05:19:47.586 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=23274 2024-09-13T05:19:47.586 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd21435 2024-09-13T05:19:47.586 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:47.586 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:47.587 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:47.588 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:47.589 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:47.589 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:47.589 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd21435 2024-09-13T05:19:47.594 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:47.594+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:47.670 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:47.670 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23274\b' 2024-09-13T05:19:47.670 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23274\b' 2024-09-13T05:19:47.674 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:47.674+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:47.676 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:47.676 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:47.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:47.716 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:47.716 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:47.716 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *23274\b' 2024-09-13T05:19:47.717 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *23274\b' 2024-09-13T05:19:47.723 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:47.723 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:47.776 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:47.777+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:47.862 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:47.862 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:47.877 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:47.874+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:47.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:47.978 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:47.978+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:48.045 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:48.045 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:48.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:48.102 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:48.102+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:48.139 INFO:teuthology.orchestra.run.smithi016.stderr:set 3.3 pg_upmap_items mapping to [1->0,4->5] 2024-09-13T05:19:48.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:19:48.206 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:48.205+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:48.224 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:48.224 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config global set global rbd_default_pool new_default_pool21435 2024-09-13T05:19:48.234 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:48.234 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:48.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:48.275 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd create --size 10M testrbdnbd21435 2024-09-13T05:19:48.294 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:48.293+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:48.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:48.343 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd21435 2024-09-13T05:19:48.344 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:48.344 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:48.345 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:48.345 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:48.347 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:48.347 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:48.347 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd21435 2024-09-13T05:19:48.382 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:48.381+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:48.412 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:48.412 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:48.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:48.473 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:48.483 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:48.477+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:48.490 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:48.492 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:48.492 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid new_default_pool21435 2024-09-13T05:19:48.492 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=new_default_pool21435 2024-09-13T05:19:48.492 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:48.493 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:48.493 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''new_default_pool21435'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:48.507 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:48.527 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:48.548 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:48.548 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:48.554 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23400 2024-09-13T05:19:48.554 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23400 2024-09-13T05:19:48.554 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23400 -C rbd-nbd 2024-09-13T05:19:48.561 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:48.562 INFO:tasks.workunit.client.0.smithi122.stdout: 23400 ? 00:00:00 rbd-nbd 2024-09-13T05:19:48.562 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device testrbdnbd21435 23400 2024-09-13T05:19:48.562 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=testrbdnbd21435 2024-09-13T05:19:48.562 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=23400 2024-09-13T05:19:48.563 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd21435 2024-09-13T05:19:48.563 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:48.563 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:48.563 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:48.563 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:48.565 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:48.565 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:48.565 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd21435 2024-09-13T05:19:48.568 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:48.568 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:48.584 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:48.585+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:48.585 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:48.585 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:48.585 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:48.586 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:48.586 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:48.599 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:48.648 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:48.648 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23400\b' 2024-09-13T05:19:48.648 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23400\b' 2024-09-13T05:19:48.685 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:48.685+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:48.694 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:48.694 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:19:48.695 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *23400\b' 2024-09-13T05:19:48.695 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *23400\b' 2024-09-13T05:19:48.701 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:48.701 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:19:48.765 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:48.765 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:48.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:48.765 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.436025001000001, 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.766 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:19:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:19:48.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:19:48.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.424819240000001, 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:19:48.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.414640186, 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.407204383, 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:19:48.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:48.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:19:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:19:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.40059668, 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:19:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:19:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.391370462000001, 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4797.0:1 3.3 e48/47 3:c4c92e5a:::rbd_trash:head v 48'1, mlcod=0'0)", 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 25.303876548000002, 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015947019999999999, 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459810+0000", 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8649999999999998e-06 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459820+0000", 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9899999999999992e-06 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459829+0000", 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.431e-06 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459885+0000", 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5868000000000001e-05 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459912+0000", 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7203e-05 2024-09-13T05:19:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461364+0000", 2024-09-13T05:19:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014515820000000001 2024-09-13T05:19:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461399+0000", 2024-09-13T05:19:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5063e-05 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:180 2.0 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.339330416999999, 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013961119999999999, 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 180 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424351+0000", 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424350+0000", 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424354+0000", 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3560000000000003e-06 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424365+0000", 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0832999999999999e-05 2024-09-13T05:19:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424374+0000", 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5599999999999994e-06 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.424997+0000", 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00062386799999999997 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.425131+0000", 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133614 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.425747+0000", 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00061550899999999998 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:494 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.856467+0000", 2024-09-13T05:19:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.907213853000002, 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001401575, 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 494 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856467+0000", 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856467+0000", 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856466+0000", 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856473+0000", 2024-09-13T05:19:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4839999999999998e-06 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856493+0000", 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9819000000000001e-05 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.856509+0000", 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5333e-05 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.857232+0000", 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00072378299999999998 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.857354+0000", 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121625 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.857869+0000", 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00051458799999999996 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.896661062, 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:19:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:19:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:527 2.0 2:0158b2c3:::rbd_data.11d2ef348a4.000000000000000e:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.864000573999999, 2024-09-13T05:19:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001391878, 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 527 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899680+0000", 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899682+0000", 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.198e-06 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899687+0000", 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8760000000000004e-06 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.899691+0000", 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6509999999999999e-06 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.900302+0000", 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00061118999999999995 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.900505+0000", 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000203523 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.901072+0000", 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00056678000000000002 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4927.0:12 2.3 e56/16 2:d073633c:::rbd_header.11d2ef348a4:head v 56'122, mlcod=55'120)", 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.992223+0000", 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.771458060000001, 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014957080000000001, 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992223+0000", 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992223+0000", 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992221+0000", 2024-09-13T05:19:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992237+0000", 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6656000000000001e-05 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992270+0000", 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2863e-05 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992285+0000", 2024-09-13T05:19:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5026e-05 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992393+0000", 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010806099999999999 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.992458+0000", 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4258999999999996e-05 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.993654+0000", 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011958209999999999 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.993719+0000", 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5055000000000006e-05 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4883.0:22 2.3 e60/16 2:d073633c:::rbd_header.11d2ef348a4:head v 60'166, mlcod=59'164)", 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:40.106376+0000", 2024-09-13T05:19:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.6573047840000008, 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015055159999999999, 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106376+0000", 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106376+0000", 2024-09-13T05:19:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106376+0000", 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106390+0000", 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4263999999999999e-05 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106471+0000", 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1422999999999996e-05 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106485+0000", 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4081e-05 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106670+0000", 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018492899999999999 2024-09-13T05:19:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.106731+0000", 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0625000000000003e-05 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.107839+0000", 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001108277 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.107882+0000", 2024-09-13T05:19:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2443e-05 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4889.0:68 2.3 e60/16 2:d073633c:::rbd_header.11d2ef348a4:head v 60'172, mlcod=60'170)", 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:40.432192+0000", 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.3314887580000008, 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001408795, 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432192+0000", 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432192+0000", 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432190+0000", 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432207+0000", 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6393e-05 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432238+0000", 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0923000000000002e-05 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432252+0000", 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4032999999999999e-05 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432376+0000", 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012397800000000001 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.432431+0000", 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5798999999999999e-05 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.433538+0000", 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00110614 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:40.433601+0000", 2024-09-13T05:19:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3441999999999993e-05 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:48 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 69632~61440] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999956+0000", 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7637249260000001, 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00139817, 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:19:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999956+0000", 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999956+0000", 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999954+0000", 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999958+0000", 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4190000000000004e-06 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999965+0000", 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5529999999999998e-06 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999979+0000", 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3784000000000001e-05 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001059+0000", 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010799990000000001 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001176+0000", 2024-09-13T05:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116995 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001354+0000", 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017862999999999999 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7636982029999997, 2024-09-13T05:19:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:19:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:19:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4982.0:50 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:5 [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.88870439, 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001569698, 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4982", 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1259763078", 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 50 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874976+0000", 2024-09-13T05:19:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967296 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874978+0000", 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2100000000000001e-06 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874981+0000", 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1760000000000002e-06 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874984+0000", 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9569999999999998e-06 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876017+0000", 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00103304 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876260+0000", 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000243403 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876546+0000", 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028614500000000002 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 62/16 rep_tid 413 entries 62'176 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5289.0:78 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 60'174)", 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.946129+0000", 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.8175518369999999, 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014672419999999999, 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.946129+0000", 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.946129+0000", 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.946127+0000", 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.946150+0000", 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3226000000000001e-05 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.946205+0000", 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4926000000000002e-05 2024-09-13T05:19:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.946256+0000", 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0586e-05 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.946434+0000", 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017853699999999999 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.947596+0000", 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011621699999999999 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:48 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 69632~61440] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978517+0000", 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.7851634289999998, 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001385144, 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:19:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978517+0000", 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978517+0000", 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978517+0000", 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978521+0000", 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1239999999999998e-06 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978530+0000", 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978537+0000", 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979600+0000", 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010631519999999999 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979721+0000", 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012145400000000001 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979903+0000", 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018127400000000001 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.785137449, 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017381950000000001, 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978546+0000", 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2059999999999999e-06 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978555+0000", 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5969999999999992e-06 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978561+0000", 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7390000000000002e-06 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979976+0000", 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014147669999999999 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980108+0000", 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013226399999999999 2024-09-13T05:19:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980282+0000", 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017327799999999999 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:48.860 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:48.862 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:48.862 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:48.861+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:48.963 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:48.961+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:48.992 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:48.992 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:48.992 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:48.993 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:49.006 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:49.064 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:49.065+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:49.130 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:49.130 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:49.130 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:49.130 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:49.130 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:49.144 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:49.187 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:49.189+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:49.202 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:19:49.203 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config global rm global rbd_default_pool 2024-09-13T05:19:49.257 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:49.257 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:49.257 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:49.258 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:49.258 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:49.259 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:49.259 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:49.259 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:19:49.266 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:49.265+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:49.276 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:19:49.315 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:49.315 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:49.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:49.315 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:49.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.315 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:49.315 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:49.315 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.180581024000006, 2024-09-13T05:19:49.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:49.319 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.179949765000003, 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.320 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:49.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:49.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.324 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.179304643999998, 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:49.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:49.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.178797281000001, 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.329 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:49.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.333 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.178271844999998, 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:49.334 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.335 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:49.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:49.338 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.177634333, 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.340 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.341 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:49.342 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.177057625000003, 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.343 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.344 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.345 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:49.346 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.176506152000002, 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.349 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.350 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:49.351 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:49.352 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.176045568999996, 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:49.353 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.354 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.355 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:49.356 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.175535683000007, 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.357 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.358 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.359 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.360 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.361 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 99.175108620000003, 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.362 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.364 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.365 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.365 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:49.365 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:49.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.365 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.365 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.365 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:49.365 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:49.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.366 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 98.788348901999996, 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.368 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.369 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:49.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.371 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:49.372 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 98.170814089999993, 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.373 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:49.377 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 27.940035155, 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.378 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:19:49.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:19:49.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 27.929414811000001, 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.383 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.384 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:19:49.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:49.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.389 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 27.922200496999999, 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.390 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.391 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:19:49.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:19:49.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.395 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 27.915844976999999, 2024-09-13T05:19:49.396 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:49.397 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:19:49.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:19:49.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.402 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 27.915788613, 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.403 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:19:49.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:19:49.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:19:49.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.409 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 27.915751042, 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.410 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:19:49.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:19:49.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:49.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:19:49.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:49.416 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 27.915716518, 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:49.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:49.420 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.422 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.423 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.423 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.423 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:49.425 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:49.425 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:49.425 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:49.425 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:49.425 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:49.425 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:49.426 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:49.426 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:49.425+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:49.460 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23494 2024-09-13T05:19:49.460 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23494 2024-09-13T05:19:49.461 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23494 -C rbd-nbd 2024-09-13T05:19:49.468 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:49.468 INFO:tasks.workunit.client.0.smithi122.stdout: 23494 ? 00:00:00 rbd-nbd 2024-09-13T05:19:49.468 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo kill 23494 2024-09-13T05:19:49.469 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:49.469 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:49.470 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:49.470 INFO:tasks.workunit.client.0.smithi122.stderr:++ which kill 2024-09-13T05:19:49.470 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/kill 2024-09-13T05:19:49.471 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:49.471 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/kill 23494 2024-09-13T05:19:49.487 INFO:tasks.workunit.client.0.smithi122.stderr:++ seq 10 2024-09-13T05:19:49.491 INFO:tasks.workunit.client.0.smithi122.stderr:+ for i in `seq 10` 2024-09-13T05:19:49.492 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:49.492 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23494 *rbd *testrbdnbd21435' 2024-09-13T05:19:49.492 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23494 *rbd *testrbdnbd21435' 2024-09-13T05:19:49.525 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:49.525+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:49.543 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:49.543 INFO:tasks.workunit.client.0.smithi122.stderr:+ break 2024-09-13T05:19:49.544 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:19:49.544 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23494 *rbd *testrbdnbd21435' 2024-09-13T05:19:49.544 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23494 *rbd *testrbdnbd21435' 2024-09-13T05:19:49.590 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:49.590 INFO:tasks.workunit.client.0.smithi122.stderr:+ QUIESCE_HOOK=/tmp/tmp.WvgaeitFGU/quiesce.sh 2024-09-13T05:19:49.591 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.WvgaeitFGU/quiesce.sh rbd/testrbdnbd21435 2024-09-13T05:19:49.591 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:19:49.591 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:19:49.592 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:19:49.592 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:19:49.593 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:49.593 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:49.593 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:49.593 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:49.594 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:19:49.594 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:19:49.594 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.WvgaeitFGU/quiesce.sh rbd/testrbdnbd21435 2024-09-13T05:19:49.607 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:49.651 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:49.653+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:49.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:49.725+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:49.735 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:19:49.735 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:19:49.735 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:19:49.735 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:19:49.736 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:19:49.736 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:19:49.767 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:49.767 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:49.767 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:49.781 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:49.800 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23556 2024-09-13T05:19:49.800 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23556 2024-09-13T05:19:49.801 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23556 -C rbd-nbd 2024-09-13T05:19:49.808 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:19:49.808 INFO:tasks.workunit.client.0.smithi122.stdout: 23556 ? 00:00:00 rbd-nbd 2024-09-13T05:19:49.808 INFO:tasks.workunit.client.0.smithi122.stderr:+ test '!' -e /tmp/tmp.WvgaeitFGU/quiesce.sh 2024-09-13T05:19:49.809 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd snap create rbd/testrbdnbd21435@quiesce1 2024-09-13T05:19:49.809 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd21435@quiesce1 2024-09-13T05:19:49.827 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:49.829+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:49.871 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete...2024-09-13T05:19:49.866+0000 7fc9ce6c1640 -1 librbd::ImageWatcher: 0x7fc9ac0075a0 failed to notify quiesce: (1) Operation not permitted 2024-09-13T05:19:49.872 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:19:49.866+0000 7fc9ce6c1640 -1 librbd::SnapshotCreateRequest: failed to notify quiesce: (1) Operation not permitted 2024-09-13T05:19:49.873 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete...failed. 2024-09-13T05:19:49.873 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: failed to create snapshot: (1) Operation not permitted 2024-09-13T05:19:49.887 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:49.887 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-13T05:19:49.887 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:49.888 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:49.889 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:49.889 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:19:49.890 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:19:49.891 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:49.891 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-13T05:19:49.906 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.566084546999999, 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.907 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:49.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:19:49.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.554203945000001, 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.913 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:19:49.914 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.915 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.548260246000002, 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:19:49.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.545584538, 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:19:49.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.926 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.540870505000001, 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:19:49.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:19:49.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.524918232000001, 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:19:49.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.515278030000001, 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:19:49.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:19:49.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.508185224999998, 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.940 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:19:49.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:19:49.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.944 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.499967268999999, 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:19:49.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:19:49.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.497821456, 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:19:49.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:19:49.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:19:49.953 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.495903662, 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:49.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:19:49.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 22.342457400000001, 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:49.958 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:19:49.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 22.336019459999999, 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.962 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:19:49.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:19:49.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.966 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 22.329951307000002, 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.967 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:49.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:19:49.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.971 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 22.322407754, 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:19:49.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 22.309730509000001, 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:49.976 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:49.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:19:49.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:19:49.980 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 13.823106642999999, 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:19:49.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:19:49.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:19:49.985 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 13.823037822, 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:19:49.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 9.7946352519999991, 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.990 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:19:49.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:19:49.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:49.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:49.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 1.2679549379999999, 2024-09-13T05:19:49.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:19:49.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:49.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:50.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:19:50.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:19:50.002 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.002 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.002 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.002 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.002 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.002 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:50.004 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-09-13T05:19:50.004 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-09-13T05:19:50.004 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386343 s, 27.1 MB/s 2024-09-13T05:19:50.004 INFO:tasks.workunit.client.0.smithi122.stderr:+ touch /tmp/tmp.WvgaeitFGU/quiesce.sh 2024-09-13T05:19:50.004 INFO:tasks.workunit.client.0.smithi122.stderr:+ chmod +x /tmp/tmp.WvgaeitFGU/quiesce.sh 2024-09-13T05:19:50.004 INFO:tasks.workunit.client.0.smithi122.stderr:+ cat 2024-09-13T05:19:50.004 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd21435@quiesce1 2024-09-13T05:19:50.005 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:50.005 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:50.005+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:50.105 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:50.105+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:50.175 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:50.175 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:50.175 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:50.175 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:50.190 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:50.235 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:50.233+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:50.306 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:50.305+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:50.308 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T05:19:50.316 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-13T05:19:50.316 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:50.317 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:50.317 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:50.318 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:19:50.318 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:19:50.319 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:50.319 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-13T05:19:50.343 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-09-13T05:19:50.344 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-09-13T05:19:50.344 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019152 s, 54.8 MB/s 2024-09-13T05:19:50.345 INFO:tasks.workunit.client.0.smithi122.stderr:++ cat /tmp/tmp.WvgaeitFGU/quiesce 2024-09-13T05:19:50.347 INFO:tasks.workunit.client.0.smithi122.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-09-13T05:19:50.347 INFO:tasks.workunit.client.0.smithi122.stderr:++ cat /tmp/tmp.WvgaeitFGU/unquiesce 2024-09-13T05:19:50.348 INFO:tasks.workunit.client.0.smithi122.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-09-13T05:19:50.348 INFO:tasks.workunit.client.0.smithi122.stderr:+ touch /tmp/tmp.WvgaeitFGU/quiesce.sh 2024-09-13T05:19:50.349 INFO:tasks.workunit.client.0.smithi122.stderr:+ chmod +x /tmp/tmp.WvgaeitFGU/quiesce.sh 2024-09-13T05:19:50.350 INFO:tasks.workunit.client.0.smithi122.stderr:+ cat 2024-09-13T05:19:50.351 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false rbd snap create rbd/testrbdnbd21435@quiesce2 2024-09-13T05:19:50.351 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd21435@quiesce2 2024-09-13T05:19:50.364 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:50.365 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:50.365 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:50.365 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:50.365 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:50.379 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:50.407 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:50.405+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:50.423 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 10% complete...failed. 2024-09-13T05:19:50.423 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: failed to create snapshot: (22) Invalid argument 2024-09-13T05:19:50.432 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:19:50.432 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-13T05:19:50.432 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:50.432 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:50.433 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:50.433 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:19:50.435 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:19:50.435 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:50.435 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-13T05:19:50.454 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-09-13T05:19:50.454 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-09-13T05:19:50.455 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130512 s, 80.3 MB/s 2024-09-13T05:19:50.455 INFO:tasks.workunit.client.0.smithi122.stderr:+ cat 2024-09-13T05:19:50.456 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd21435@quiesce2 2024-09-13T05:19:50.508 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:50.508 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:50.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:50.508 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:50.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.508 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:19:50.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:50.508 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.460082645, 2024-09-13T05:19:50.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:50.513 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.423682067, 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:19:50.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.420098763, 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:50.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:19:50.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:19:50.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.415464676999999, 2024-09-13T05:19:50.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:19:50.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.526 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.412227400999999, 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:19:50.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.530 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.408548002, 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:19:50.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.405191809, 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:19:50.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:19:50.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:19:50.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:19:50.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.398860515999999, 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.539 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:19:50.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:19:50.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:19:50.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:19:50.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:19:50.543 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.543 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.543 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.544 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:50.544 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:50.544 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.391180566999999, 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:50.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:19:50.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.388549423000001, 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:19:50.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:19:50.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:19:50.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:19:50.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.554 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.384707463, 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:19:50.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.380872067, 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:19:50.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.377416821000001, 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:50.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:19:50.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.566 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.373342981, 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:19:50.567 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:19:50.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.571 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.370264052, 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:19:50.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:19:50.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:19:50.576 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.576 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.576 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.576 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:50.576 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.366021035999999, 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:19:50.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:19:50.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:19:50.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:19:50.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:19:50.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:19:50.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:19:50.582 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.582 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.582 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.582 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:50.583 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:50.583 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.362145033999999, 2024-09-13T05:19:50.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:19:50.583 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.583 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.583 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:50.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:50.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:19:50.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:50.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:19:50.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:19:50.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:19:50.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:19:50.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:19:50.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:19:50.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:19:50.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:19:50.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:19:50.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:19:50.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:19:50.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:19:50.591 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.591 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.591 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.592 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.592 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:50.592 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.354519409, 2024-09-13T05:19:50.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:19:50.592 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.593 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.593 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:50.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:50.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:19:50.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:50.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:19:50.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:19:50.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:19:50.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:19:50.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:19:50.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:19:50.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:19:50.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:19:50.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:19:50.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:19:50.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.600 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:19:50.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:19:50.601 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.601 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.601 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.351252999, 2024-09-13T05:19:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:19:50.603 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.603 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.603 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:50.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:50.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:19:50.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:19:50.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.606 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.606 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:19:50.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:19:50.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.607 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:19:50.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:19:50.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:19:50.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:19:50.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:19:50.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:19:50.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:19:50.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:19:50.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:19:50.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.347560595999999, 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:50.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:50.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:50.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:50.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:50.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:19:50.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:19:50.615 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.615 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.615 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.615 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:50.615 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:50.615 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:50.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:50.617 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:50.617+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:50.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:50.718 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:50.717+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:50.756 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:50.757 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:50.757 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:50.757 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:50.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:50.780 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:50.797 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:50.814 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:50.814 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:50.819 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:50.817+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:50.834 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:50.834 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:50.854 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:50.875 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:50.876 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:50.898 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:50.898 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:50.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:50.958 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:50.957+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:51.020 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:51.017+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:51.088 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:51.089 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:51.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:51.146 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:51.145+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:51.222 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:51.221+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:51.274 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:51.274 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:51.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:51.302 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:19:51.338 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:51.337+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:51.423 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:51.425+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:51.455 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:51.455 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:51.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:51.523 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:51.521+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:51.587 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:51.587 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:51.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:51.650 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:51.649+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:51.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:51.721+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:51.768 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:51.768 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:51.783 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:51.826 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:51.825+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:51.926 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:51.925+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:51.951 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:51.951 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:51.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:52.050 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:52.049+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:52.127 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:52.129+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:52.132 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:52.132 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:52.147 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:52.228 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:52.225+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:52.279 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:52.279 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:52.294 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:19:52.296 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:52.358 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:52.357+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:52.412 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:52.426 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:52.430 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:52.429+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:52.530 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:52.529+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:52.546 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:52.546 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:52.560 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:52.651 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:52.653+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:52.731 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:52.732 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:52.732 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:52.729+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:52.746 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:52.833 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:52.829+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:52.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:52.903 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:52.903 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:52.917 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:52.938 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:52.937+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:53.034 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:53.033+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:53.036 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:53.036 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:53.047 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:53.048 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:53.050 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:53.064 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:53.064 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:53.081 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:53.100 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:53.100 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:53.118 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:53.135 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:53.136 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:53.133+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:53.136 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:53.162 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:19:53.163 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:19:53.163 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T05:19:53.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T05:19:53.214 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:53.215 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:53.215 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:53.215 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:53.230 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:53.275 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:53.277+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:53.337 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:53.333+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:53.354 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:53.354 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:53.355 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:53.355 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:53.355 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:53.368 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:53.455 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:53.457+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:19:53.496 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 67, 2024-09-13T05:19:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5", 2024-09-13T05:19:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2024-09-13T05:17:57.363809+0000", 2024-09-13T05:19:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2024-09-13T05:19:52.281088+0000", 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2024-09-13T05:19:34.047702+0000", 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2024-09-13T05:17:59.738330+0000", 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5806080, 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "nodeep-scrub", 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "noscrub", 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 7, 2024-09-13T05:19:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 3, 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "reef", 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_crimson": false, 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": ".mgr", 2024-09-13T05:19:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:08.122164+0000", 2024-09-13T05:19:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-13T05:19:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2024-09-13T05:19:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:19:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:19:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:19:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:19:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:19:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:19:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:19:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "15", 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2024-09-13T05:19:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:19:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:19:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": {} 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 6.059999942779541, 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 6.059999942779541, 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.33000001311302185, 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2, 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2, 2024-09-13T05:19:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:15.001664+0000", 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:19:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:19:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "66", 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:19:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 6, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 66, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:19:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:19:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 2.25, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 2.25, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 1, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2.25, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2.25, 2024-09-13T05:19:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 3, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "new_default_pool21435", 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:19:21.793970+0000", 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:19:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:19:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "50", 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 3, 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 50, 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:19:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:19:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 3.0099999904632568, 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 3.0099999904632568, 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.82999998331069946, 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2.5, 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2.5, 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "e58174f4-85ec-4321-8bc3-384261c59c6b", 2024-09-13T05:19:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 48, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-13T05:19:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2024-09-13T05:19:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/2153409514", 2024-09-13T05:19:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/2153409514", 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/2153409514", 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/2153409514", 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "84a3db6c-dacd-43d5-b964-37fdef0091a4", 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 54, 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2024-09-13T05:19:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.526 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/2608485151", 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/2608485151", 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/2608485151", 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/2608485151", 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "0d530476-cf38-4544-9ce7-e8ed35df220b", 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 47, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.529 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2024-09-13T05:19:53.530 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:53.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/1006404377", 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/1006404377", 2024-09-13T05:19:53.532 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/1006404377", 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/1006404377", 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "fd91f7e2-580d-47ea-9db5-792c8f1ebe01", 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:53.533 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.980804443359375, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 20, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 29, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 47, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 21, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6816", 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6817", 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6818", 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6819", 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6822", 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6823", 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6820", 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.537 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6821", 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6817/1834661236", 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6819/1834661236", 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6823/1834661236", 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6821/1834661236", 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.538 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "01af0f4c-5da2-46fe-a262-59fcc6a2efa9", 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 38, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 54, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 54, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 40, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6808", 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6809", 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6810", 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6811", 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6814", 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6815", 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6812", 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6813", 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6809/3457694380", 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6811/3457694380", 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6815/3457694380", 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6813/3457694380", 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:53.543 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "64cdd5f2-1568-4e96-ad90-6ef7414651f0", 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:19:53.544 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 47, 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6800", 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.545 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6801", 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6802", 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6803", 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6806", 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6807", 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6804", 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6805", 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6801/2778785304", 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6803/2778785304", 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6807/2778785304", 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6805/2778785304", 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:19:53.549 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.193869+0000", 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.255764+0000", 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:53.551 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.286354+0000", 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.221310+0000", 2024-09-13T05:19:53.552 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 21 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.026235+0000", 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 40 2024-09-13T05:19:53.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.010242+0000", 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:19:53.554 INFO:teuthology.orchestra.run.smithi016.stdout: 0, 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_primaries": [], 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2024-09-13T05:19:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2024-09-13T05:19:53.556 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2024-09-13T05:19:53.557 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2024-09-13T05:19:53.557 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2024-09-13T05:19:53.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:53.557 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:53.558 INFO:tasks.thrashosds.thrasher:j is {'epoch': 67, 'fsid': '1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5', 'created': '2024-09-13T05:17:57.363809+0000', 'modified': '2024-09-13T05:19:52.281088+0000', 'last_up_change': '2024-09-13T05:19:34.047702+0000', 'last_in_change': '2024-09-13T05:17:59.738330+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T05:18:08.122164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T05:18:15.001664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 66, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool21435', 'create_time': '2024-09-13T05:19:21.793970+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 50, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e58174f4-85ec-4321-8bc3-384261c59c6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 2153409514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 2153409514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 2153409514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 2153409514}]}, 'public_addr': '172.21.15.16:6801/2153409514', 'cluster_addr': '172.21.15.16:6803/2153409514', 'heartbeat_back_addr': '172.21.15.16:6807/2153409514', 'heartbeat_front_addr': '172.21.15.16:6805/2153409514', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '84a3db6c-dacd-43d5-b964-37fdef0091a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 2608485151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 2608485151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 2608485151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 2608485151}]}, 'public_addr': '172.21.15.16:6809/2608485151', 'cluster_addr': '172.21.15.16:6811/2608485151', 'heartbeat_back_addr': '172.21.15.16:6815/2608485151', 'heartbeat_front_addr': '172.21.15.16:6813/2608485151', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0d530476-cf38-4544-9ce7-e8ed35df220b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 1006404377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 1006404377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 1006404377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 1006404377}]}, 'public_addr': '172.21.15.16:6817/1006404377', 'cluster_addr': '172.21.15.16:6819/1006404377', 'heartbeat_back_addr': '172.21.15.16:6823/1006404377', 'heartbeat_front_addr': '172.21.15.16:6821/1006404377', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fd91f7e2-580d-47ea-9db5-792c8f1ebe01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.980804443359375, 'last_clean_begin': 12, 'last_clean_end': 20, 'up_from': 29, 'up_thru': 47, 'down_at': 21, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6816', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6817', 'nonce': 1834661236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6818', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6819', 'nonce': 1834661236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6822', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6823', 'nonce': 1834661236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6820', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6821', 'nonce': 1834661236}]}, 'public_addr': '172.21.15.79:6817/1834661236', 'cluster_addr': '172.21.15.79:6819/1834661236', 'heartbeat_back_addr': '172.21.15.79:6823/1834661236', 'heartbeat_front_addr': '172.21.15.79:6821/1834661236', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '01af0f4c-5da2-46fe-a262-59fcc6a2efa9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 12, 'last_clean_end': 38, 'up_from': 54, 'up_thru': 54, 'down_at': 40, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6808', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6809', 'nonce': 3457694380}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6810', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6811', 'nonce': 3457694380}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6814', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6815', 'nonce': 3457694380}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6812', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6813', 'nonce': 3457694380}]}, 'public_addr': '172.21.15.79:6809/3457694380', 'cluster_addr': '172.21.15.79:6811/3457694380', 'heartbeat_back_addr': '172.21.15.79:6815/3457694380', 'heartbeat_front_addr': '172.21.15.79:6813/3457694380', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '64cdd5f2-1568-4e96-ad90-6ef7414651f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6800', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6801', 'nonce': 2778785304}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6802', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6803', 'nonce': 2778785304}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6806', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6807', 'nonce': 2778785304}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6804', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6805', 'nonce': 2778785304}]}, 'public_addr': '172.21.15.79:6801/2778785304', 'cluster_addr': '172.21.15.79:6803/2778785304', 'heartbeat_back_addr': '172.21.15.79:6807/2778785304', 'heartbeat_front_addr': '172.21.15.79:6805/2778785304', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.193869+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.255764+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.286354+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.221310+0000', 'dead_epoch': 21}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.026235+0000', 'dead_epoch': 40}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.010242+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [0, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T05:19:53.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:19:53.560 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.492109134, 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:19:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.455708556000001, 2024-09-13T05:19:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:19:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:19:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.452125251999998, 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:19:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:19:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.447491165999999, 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:19:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:19:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.444253889999999, 2024-09-13T05:19:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:19:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:19:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:19:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.440574491, 2024-09-13T05:19:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:19:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.437218298000001, 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:19:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:19:53.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.430887004999999, 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.589 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:53.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:19:53.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.423207055999999, 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:19:53.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.420575912, 2024-09-13T05:19:53.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:19:53.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.600 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:19:53.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:19:53.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.600 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:19:53.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.416733952000001, 2024-09-13T05:19:53.602 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:19:53.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.412898556000002, 2024-09-13T05:19:53.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.607 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:19:53.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.40944331, 2024-09-13T05:19:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:19:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.612 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.40536947, 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.615 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:19:53.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.402290540999999, 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:19:53.619 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:19:53.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:19:53.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.398047524999999, 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.624 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:53.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.625 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.625 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:19:53.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:19:53.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:53.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:19:53.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:19:53.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:19:53.628 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:19:53.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.628 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:19:53.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:19:53.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:19:53.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:19:53.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:19:53.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:19:53.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:19:53.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:19:53.631 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.631 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.632 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.632 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:19:53.632 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:53.633 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.394171523000001, 2024-09-13T05:19:53.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:19:53.633 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.633 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.633 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:53.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:19:53.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:19:53.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:53.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.636 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:19:53.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:19:53.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:19:53.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:19:53.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:19:53.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:19:53.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:19:53.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:19:53.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:19:53.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:19:53.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:19:53.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:19:53.641 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.641 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.641 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.642 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.642 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:53.642 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.386545898000001, 2024-09-13T05:19:53.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:19:53.642 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.642 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.643 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:53.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:19:53.644 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.645 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:19:53.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:53.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:19:53.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:19:53.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:19:53.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:19:53.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:19:53.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:19:53.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:19:53.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:19:53.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:19:53.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:19:53.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:19:53.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:19:53.651 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.651 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.651 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.651 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.651 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:53.652 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.383279487999999, 2024-09-13T05:19:53.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:19:53.652 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.652 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.652 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:53.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:19:53.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 18.379587085000001, 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:19:53.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:19:53.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.660 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:53.661 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:53.661 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:53.661 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:53.557+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:53.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:53.663 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:53.661+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:53.764 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:53.765+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:53.788 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:53.789 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:53.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:53.864 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:53.865+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:53.922 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:53.922 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:53.922 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:53.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:53.965 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:53.965+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:19:53.976 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:19:53.998 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-13T05:19:52.291231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134808317,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":3586,"num_read_kb":172511,"num_write":715,"num_write_kb":264748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":664,"ondisk_log_size":664,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":456488,"kb_used_data":267624,"kb_used_omap":14,"kb_used_meta":188785,"kb_avail":565774552,"statfs":{"total":579820584960,"available":579353141248,"internally_reserved":0,"allocated":274046976,"data_stored":271261120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":193315958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":569,"num_read_kb":7664,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.217748"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":206,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:43.135513+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:19:43.135513+0000","last_peered":"2024-09-13T05:19:43.135513+0000","last_clean":"2024-09-13T05:19:43.135513+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:19:43.135513+0000","last_undegraded":"2024-09-13T05:19:43.135513+0000","last_fullsized":"2024-09-13T05:19:43.135513+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"63'179","reported_seq":1053,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-13T05:19:47.532707+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:19:47.532707+0000","last_peered":"2024-09-13T05:19:47.532707+0000","last_clean":"2024-09-13T05:19:47.532707+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:19:47.532707+0000","last_undegraded":"2024-09-13T05:19:47.532707+0000","last_fullsized":"2024-09-13T05:19:47.532707+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":12582950,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1780,"num_read_kb":16091,"num_write":189,"num_write_kb":20617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"64'5","reported_seq":49,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T05:19:49.325749+0000","last_change":"2024-09-13T05:19:25.480800+0000","last_active":"2024-09-13T05:19:49.325749+0000","last_peered":"2024-09-13T05:19:49.325749+0000","last_clean":"2024-09-13T05:19:49.325749+0000","last_became_active":"2024-09-13T05:19:23.455354+0000","last_became_peered":"2024-09-13T05:19:23.455354+0000","last_unstale":"2024-09-13T05:19:49.325749+0000","last_undegraded":"2024-09-13T05:19:49.325749+0000","last_fullsized":"2024-09-13T05:19:49.325749+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:20.817288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017913759999999999,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":27,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"57'58","reported_seq":323,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:45.983078+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:19:45.983078+0000","last_peered":"2024-09-13T05:19:45.983078+0000","last_clean":"2024-09-13T05:19:45.983078+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:19:45.983078+0000","last_undegraded":"2024-09-13T05:19:45.983078+0000","last_fullsized":"2024-09-13T05:19:45.983078+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":25165824,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":207,"num_read_kb":27196,"num_write":57,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:43.269373+0000","last_change":"2024-09-13T05:19:35.075887+0000","last_active":"2024-09-13T05:19:43.269373+0000","last_peered":"2024-09-13T05:19:43.269373+0000","last_clean":"2024-09-13T05:19:43.269373+0000","last_became_active":"2024-09-13T05:19:35.073898+0000","last_became_peered":"2024-09-13T05:19:35.073898+0000","last_unstale":"2024-09-13T05:19:43.269373+0000","last_undegraded":"2024-09-13T05:19:43.269373+0000","last_fullsized":"2024-09-13T05:19:43.269373+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:23.211903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010913139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"60'21","reported_seq":136,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T05:19:49.325605+0000","last_change":"2024-09-13T05:18:43.272711+0000","last_active":"2024-09-13T05:19:49.325605+0000","last_peered":"2024-09-13T05:19:49.325605+0000","last_clean":"2024-09-13T05:19:49.325605+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:19:49.325605+0000","last_undegraded":"2024-09-13T05:19:49.325605+0000","last_fullsized":"2024-09-13T05:19:49.325605+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:27.589949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388662,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":64,"num_read_kb":8202,"num_write":21,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:44.625655+0000","last_change":"2024-09-13T05:19:25.538085+0000","last_active":"2024-09-13T05:19:44.625655+0000","last_peered":"2024-09-13T05:19:44.625655+0000","last_clean":"2024-09-13T05:19:44.625655+0000","last_became_active":"2024-09-13T05:19:23.462578+0000","last_became_peered":"2024-09-13T05:19:23.462578+0000","last_unstale":"2024-09-13T05:19:44.625655+0000","last_undegraded":"2024-09-13T05:19:44.625655+0000","last_fullsized":"2024-09-13T05:19:44.625655+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:48.193321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056272899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"57'56","reported_seq":322,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-13T05:19:47.459810+0000","last_change":"2024-09-13T05:19:24.517596+0000","last_active":"2024-09-13T05:19:47.459810+0000","last_peered":"2024-09-13T05:19:47.459810+0000","last_clean":"2024-09-13T05:19:47.459810+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:19:47.459810+0000","last_undegraded":"2024-09-13T05:19:47.459810+0000","last_fullsized":"2024-09-13T05:19:47.459810+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":184,"num_read_kb":36982,"num_write":56,"num_write_kb":36866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"48'1","reported_seq":30,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:43.272072+0000","last_change":"2024-09-13T05:19:25.535626+0000","last_active":"2024-09-13T05:19:43.272072+0000","last_peered":"2024-09-13T05:19:43.272072+0000","last_clean":"2024-09-13T05:19:43.272072+0000","last_became_active":"2024-09-13T05:19:23.458938+0000","last_became_peered":"2024-09-13T05:19:23.458938+0000","last_unstale":"2024-09-13T05:19:43.272072+0000","last_undegraded":"2024-09-13T05:19:43.272072+0000","last_fullsized":"2024-09-13T05:19:43.272072+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:28:08.756172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000291315,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:43.124152+0000","last_change":"2024-09-13T05:19:25.474324+0000","last_active":"2024-09-13T05:19:43.124152+0000","last_peered":"2024-09-13T05:19:43.124152+0000","last_clean":"2024-09-13T05:19:43.124152+0000","last_became_active":"2024-09-13T05:19:23.455224+0000","last_became_peered":"2024-09-13T05:19:23.455224+0000","last_unstale":"2024-09-13T05:19:43.124152+0000","last_undegraded":"2024-09-13T05:19:43.124152+0000","last_fullsized":"2024-09-13T05:19:43.124152+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:43.322756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"61'62","reported_seq":221,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:44.625817+0000","last_change":"2024-09-13T05:18:43.275214+0000","last_active":"2024-09-13T05:19:44.625817+0000","last_peered":"2024-09-13T05:19:44.625817+0000","last_clean":"2024-09-13T05:19:44.625817+0000","last_became_active":"2024-09-13T05:18:43.274886+0000","last_became_peered":"2024-09-13T05:18:43.274886+0000","last_unstale":"2024-09-13T05:19:44.625817+0000","last_undegraded":"2024-09-13T05:19:44.625817+0000","last_fullsized":"2024-09-13T05:19:44.625817+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:53.129994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":16777254,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":196,"num_read_kb":16483,"num_write":78,"num_write_kb":32804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:43.269359+0000","last_change":"2024-09-13T05:19:35.163055+0000","last_active":"2024-09-13T05:19:43.269359+0000","last_peered":"2024-09-13T05:19:43.269359+0000","last_clean":"2024-09-13T05:19:43.269359+0000","last_became_active":"2024-09-13T05:19:35.161280+0000","last_became_peered":"2024-09-13T05:19:35.161280+0000","last_unstale":"2024-09-13T05:19:43.269359+0000","last_undegraded":"2024-09-13T05:19:43.269359+0000","last_fullsized":"2024-09-13T05:19:43.269359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T05:49:15.311928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092053199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"57'40","reported_seq":342,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:45.991226+0000","last_change":"2024-09-13T05:19:35.195473+0000","last_active":"2024-09-13T05:19:45.991226+0000","last_peered":"2024-09-13T05:19:45.991226+0000","last_clean":"2024-09-13T05:19:45.991226+0000","last_became_active":"2024-09-13T05:19:35.077291+0000","last_became_peered":"2024-09-13T05:19:35.077291+0000","last_unstale":"2024-09-13T05:19:45.991226+0000","last_undegraded":"2024-09-13T05:19:45.991226+0000","last_fullsized":"2024-09-13T05:19:45.991226+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:50.348812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":217,"num_read_kb":19140,"num_write":40,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:43.269291+0000","last_change":"2024-09-13T05:19:35.076046+0000","last_active":"2024-09-13T05:19:43.269291+0000","last_peered":"2024-09-13T05:19:43.269291+0000","last_clean":"2024-09-13T05:19:43.269291+0000","last_became_active":"2024-09-13T05:19:35.074375+0000","last_became_peered":"2024-09-13T05:19:35.074375+0000","last_unstale":"2024-09-13T05:19:43.269291+0000","last_undegraded":"2024-09-13T05:19:43.269291+0000","last_fullsized":"2024-09-13T05:19:43.269291+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:58.769823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089178300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"62'44","reported_seq":158,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:46.121725+0000","last_change":"2024-09-13T05:19:35.179253+0000","last_active":"2024-09-13T05:19:46.121725+0000","last_peered":"2024-09-13T05:19:46.121725+0000","last_clean":"2024-09-13T05:19:46.121725+0000","last_became_active":"2024-09-13T05:19:35.076916+0000","last_became_peered":"2024-09-13T05:19:35.076916+0000","last_unstale":"2024-09-13T05:19:46.121725+0000","last_undegraded":"2024-09-13T05:19:46.121725+0000","last_fullsized":"2024-09-13T05:19:46.121725+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:40:25.275387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777254,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":300,"num_read_kb":12518,"num_write":31,"num_write_kb":24582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-13T05:19:43.269416+0000","last_change":"2024-09-13T05:19:35.078422+0000","last_active":"2024-09-13T05:19:43.269416+0000","last_peered":"2024-09-13T05:19:43.269416+0000","last_clean":"2024-09-13T05:19:43.269416+0000","last_became_active":"2024-09-13T05:19:35.076279+0000","last_became_peered":"2024-09-13T05:19:35.076279+0000","last_unstale":"2024-09-13T05:19:43.269416+0000","last_undegraded":"2024-09-13T05:19:43.269416+0000","last_fullsized":"2024-09-13T05:19:43.269416+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:26:47.540962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012842699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"58'81","reported_seq":597,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-13T05:19:47.524724+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:19:47.524724+0000","last_peered":"2024-09-13T05:19:47.524724+0000","last_clean":"2024-09-13T05:19:47.524724+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:19:47.524724+0000","last_undegraded":"2024-09-13T05:19:47.524724+0000","last_fullsized":"2024-09-13T05:19:47.524724+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":16777231,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":492,"num_read_kb":35772,"num_write":81,"num_write_kb":40962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":27,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":89,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":16,"acting":16,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134217930,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":3440,"num_read_kb":172384,"num_write":553,"num_write_kb":262329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268492800,"data_stored":268435860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":541,"ondisk_log_size":541,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607577,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77468,"kb_used_data":50448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94294372,"statfs":{"total":96636764160,"available":96557436928,"internally_reserved":0,"allocated":51658752,"data_stored":51194521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.42299999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.67400000000000004}]},{"osd":3,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.49399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]},{"osd":4,"up_from":54,"seq":231928233988,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69776,"kb_used_data":41668,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94302064,"statfs":{"total":96636764160,"available":96565313536,"internally_reserved":0,"allocated":42668032,"data_stored":42215491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]}]},{"osd":3,"up_from":29,"seq":124554051599,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53436,"kb_used_data":25328,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94318404,"statfs":{"total":96636764160,"available":96582045696,"internally_reserved":0,"allocated":25935872,"data_stored":25448356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.86899999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.73799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.71499999999999997}]}]},{"osd":2,"up_from":12,"seq":51539607576,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72784,"kb_used_data":37576,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94299056,"statfs":{"total":96636764160,"available":96562233344,"internally_reserved":0,"allocated":38477824,"data_stored":38021206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.46500000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.38},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Sep 13 05:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.78800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.71399999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607576,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102040,"kb_used_data":66832,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94269800,"statfs":{"total":96636764160,"available":96532275200,"internally_reserved":0,"allocated":68435968,"data_stored":67971717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.64800000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.34000000000000002}]},{"osd":3,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":5,"last update":"Fri Sep 13 05:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.46300000000000002}]}]},{"osd":0,"up_from":12,"seq":51539607576,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80984,"kb_used_data":45772,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94290856,"statfs":{"total":96636764160,"available":96553836544,"internally_reserved":0,"allocated":46870528,"data_stored":46409829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.60699999999999998}]},{"osd":2,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.749},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55400000000000005}]},{"osd":3,"last update":"Fri Sep 13 05:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.879}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Sep 13 05:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.80200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46149632,"data_stored":46137416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67121152,"data_stored":67108936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37752832,"data_stored":37748774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41947136,"data_stored":41943078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:19:54.000 INFO:tasks.thrashosds.thrasher:Setting 3.6 to [1, 3] 2024-09-13T05:19:54.000 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.6', '1', '3'] 2024-09-13T05:19:54.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.6 1 3 2024-09-13T05:19:54.062 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.927493659000007, 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:19:54.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:19:54.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 95.917712128999995, 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:19:54.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:19:54.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:19:54.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:19:54.076 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.694881875999997, 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:19:54.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:19:54.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.687852280999998, 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.081 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:19:54.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:19:54.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.647069446000003, 2024-09-13T05:19:54.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.640210973000002, 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:19:54.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:19:54.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.608069012000001, 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:19:54.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:19:54.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.994107538000002, 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:54.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:19:54.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.993946311999998, 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:19:54.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:19:54.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:80 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'83, mlcod=57'80)", 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1452092540000001, 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072315670000000004, 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915606+0000", 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:54.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915894+0000", 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028823899999999998 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915934+0000", 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9659000000000001e-05 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915953+0000", 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8660000000000001e-05 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916091+0000", 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790800000000001 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916150+0000", 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9079000000000001e-05 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922698+0000", 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065481339999999997 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922839+0000", 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140805 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:81 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'84, mlcod=57'80)", 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1399987600000001, 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003098127, 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920815+0000", 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920950+0000", 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502499999999999 2024-09-13T05:19:54.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920989+0000", 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8779000000000003e-05 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921013+0000", 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3753000000000001e-05 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921422+0000", 2024-09-13T05:19:54.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040899799999999999 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921481+0000", 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8811999999999999e-05 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923815+0000", 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002333852 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923916+0000", 2024-09-13T05:19:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101032 2024-09-13T05:19:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:82 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'85, mlcod=57'80)", 2024-09-13T05:19:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.922538+0000", 2024-09-13T05:19:54.122 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1382783529999996, 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002777046, 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922538+0000", 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922538+0000", 2024-09-13T05:19:54.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922536+0000", 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922652+0000", 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116105 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922682+0000", 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9975999999999999e-05 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922701+0000", 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8159999999999999e-05 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922927+0000", 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022668300000000001 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922996+0000", 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8619000000000001e-05 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.925244+0000", 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022477859999999999 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.925315+0000", 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1313e-05 2024-09-13T05:19:54.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:85 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'88, mlcod=65'84)", 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.928222+0000", 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1325940499999998, 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026768769999999998, 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928222+0000", 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928222+0000", 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928220+0000", 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928327+0000", 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106729 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928359+0000", 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1333e-05 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928399+0000", 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0089999999999997e-05 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928588+0000", 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018953000000000001 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928651+0000", 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3182000000000006e-05 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930739+0000", 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00208741 2024-09-13T05:19:54.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930899+0000", 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016028700000000001 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:86 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'89, mlcod=65'85)", 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.930534+0000", 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1302818339999998, 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023356520000000001, 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930534+0000", 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930534+0000", 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930533+0000", 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930682+0000", 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014889100000000001 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930749+0000", 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7274999999999995e-05 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930763+0000", 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4144e-05 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930968+0000", 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020489299999999999 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931001+0000", 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3210000000000002e-05 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932772+0000", 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001770771 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932870+0000", 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8071000000000003e-05 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.135 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:159 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.302269+0000", 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.7585469890000001, 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026821850000000001, 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 159 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302269+0000", 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302269+0000", 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302265+0000", 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302287+0000", 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1353999999999999e-05 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302335+0000", 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7715e-05 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302367+0000", 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1977000000000002e-05 2024-09-13T05:19:54.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302514+0000", 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147424 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302534+0000", 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9760000000000001e-05 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.304769+0000", 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022353669999999998 2024-09-13T05:19:54.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.304849+0000", 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9797999999999997e-05 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.304951+0000", 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010253 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:169 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'92, mlcod=65'89)", 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.7303901750000001, 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004357662, 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330424+0000", 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330542+0000", 2024-09-13T05:19:54.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117475 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330570+0000", 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8036999999999999e-05 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330585+0000", 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4983e-05 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330651+0000", 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6361999999999999e-05 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330674+0000", 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3064000000000001e-05 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334660+0000", 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039862229999999997 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334784+0000", 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012329499999999999 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:170 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'93, mlcod=65'89)", 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.7251935270000001, 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035268080000000002, 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335621+0000", 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335688+0000", 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7006000000000002e-05 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335716+0000", 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.798e-05 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335728+0000", 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1712999999999999e-05 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335838+0000", 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110575 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335878+0000", 2024-09-13T05:19:54.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9969000000000003e-05 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339094+0000", 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032155399999999998 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339149+0000", 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5685e-05 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:172 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'95, mlcod=65'89)", 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.7230529240000001, 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029153970000000001, 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337762+0000", 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337854+0000", 2024-09-13T05:19:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2342999999999993e-05 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337868+0000", 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3658e-05 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337876+0000", 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5900000000000002e-06 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338010+0000", 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013445199999999999 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338063+0000", 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2762000000000001e-05 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340641+0000", 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002577663 2024-09-13T05:19:54.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340679+0000", 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8172e-05 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:174 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'97, mlcod=65'90)", 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.338917+0000", 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.7218988949999998, 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027662759999999998, 2024-09-13T05:19:54.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338917+0000", 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338917+0000", 2024-09-13T05:19:54.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338915+0000", 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339160+0000", 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024431000000000002 2024-09-13T05:19:54.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339190+0000", 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0171999999999998e-05 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339208+0000", 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8053000000000001e-05 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339628+0000", 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042042700000000002 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339691+0000", 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2212999999999995e-05 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.341629+0000", 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019388770000000001 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.341684+0000", 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4144000000000002e-05 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:238 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'101, mlcod=66'99)", 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.6117229059999998, 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048419240000000001, 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.159 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449091+0000", 2024-09-13T05:19:54.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450611+0000", 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015204629999999999 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450655+0000", 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3662999999999999e-05 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450676+0000", 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0573000000000001e-05 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450825+0000", 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014922399999999999 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450859+0000", 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3779999999999998e-05 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453864+0000", 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030051959999999999 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453935+0000", 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1302000000000003e-05 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.163 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:54.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:54.166 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:54.165+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:54.265 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:54.261+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:54.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:19:54.320 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:54.320 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:54.320 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:54.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:54.367 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:54.365+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:54.467 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:54.465+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:54.487 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:54.487 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:54.488 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:54.488 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:54.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:54.586 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:54.585+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:54.660 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:54.660 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 104.14137055800001, 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:54.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:54.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:54.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 104.141098533, 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.668 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:54.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 104.141054947, 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:54.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 104.14102272, 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:54.680 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:54.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.685 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 104.140811572, 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:54.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:54.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:54.691 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 104.140775279, 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:54.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:54.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:54.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 104.14073682900001, 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:54.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:54.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:54.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 104.14056192699999, 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.704 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:54.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:54.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.325857988999999, 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:19:54.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:19:54.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.318327875999998, 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:19:54.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.308758548999997, 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.721 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:19:54.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:19:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:19:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:19:54.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:54.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.297595676999997, 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:19:54.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:19:54.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:54.732 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.291436079999997, 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:19:54.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:19:54.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:19:54.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:19:54.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.242600299999999, 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.738 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:19:54.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:19:54.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:19:54.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.234323766000003, 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:54.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:19:54.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:19:54.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7415633579999996, 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:19:54.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:19:54.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7411282110000004, 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:19:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:19:54.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.760 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7408142189999998, 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:19:54.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:19:54.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7404089679999997, 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:54.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:19:54.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:19:54.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.772 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7400844299999996, 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:54.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:54.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:54.776 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:54.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:54.778 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:54.777+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:54.879 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:54.876+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:54.946 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:54.946 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:54.946 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:54.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:54.980 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:54.980+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:55.081 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:55.077+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:55.084 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:55.084 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:55.084 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:55.098 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:55.154 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:55.186 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:55.185+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:55.263 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:55.263 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.932767570000003, 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:19:55.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:19:55.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.921561808999996, 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:19:55.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:19:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:19:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.911382754999998, 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:19:55.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.903946951999998, 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:19:55.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:19:55.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.897339248999998, 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:19:55.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:19:55.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.888113031000003, 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4797.0:1 3.3 e48/47 3:c4c92e5a:::rbd_trash:head v 48'1, mlcod=0'0)", 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.800619117, 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015947019999999999, 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.293 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459810+0000", 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8649999999999998e-06 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459820+0000", 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9899999999999992e-06 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459829+0000", 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.431e-06 2024-09-13T05:19:55.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459885+0000", 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5868000000000001e-05 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459912+0000", 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7203e-05 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461364+0000", 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014515820000000001 2024-09-13T05:19:55.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461399+0000", 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5063e-05 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.393403630999998, 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:19:55.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:19:55.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.260440772000001, 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:19:55.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4982.0:50 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:5 [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:55.305 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.385446958999999, 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001569698, 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4982", 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1259763078", 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 50 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874976+0000", 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967296 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874978+0000", 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2100000000000001e-06 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874981+0000", 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1760000000000002e-06 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874984+0000", 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9569999999999998e-06 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876017+0000", 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00103304 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876260+0000", 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000243403 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876546+0000", 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028614500000000002 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:55.309 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.2818800180000007, 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017381950000000001, 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978546+0000", 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2059999999999999e-06 2024-09-13T05:19:55.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978555+0000", 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5969999999999992e-06 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978561+0000", 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7390000000000002e-06 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979976+0000", 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014147669999999999 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980108+0000", 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013226399999999999 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980282+0000", 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017327799999999999 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.313 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'182, mlcod=65'180)", 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.5708919439999995, 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016537559999999999, 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689530+0000", 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689546+0000", 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6439999999999998e-05 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689575+0000", 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9159999999999999e-05 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689590+0000", 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5056e-05 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689724+0000", 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133518 2024-09-13T05:19:55.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689793+0000", 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8769000000000005e-05 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691110+0000", 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001317807 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691185+0000", 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4876000000000006e-05 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5233.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'183, mlcod=65'181)", 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4030822230000002, 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001624459, 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.318 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857339+0000", 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857354+0000", 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5157000000000001e-05 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857383+0000", 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8759e-05 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857396+0000", 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3223e-05 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857503+0000", 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010699199999999999 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857575+0000", 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1092000000000003e-05 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.858894+0000", 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001319408 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.858966+0000", 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1811999999999996e-05 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5233.0:16 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'185, mlcod=65'183)", 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.864111+0000", 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.3963129170000004, 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001563947, 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864111+0000", 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864111+0000", 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864109+0000", 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864125+0000", 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.624e-05 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864159+0000", 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3846000000000001e-05 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864177+0000", 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7629999999999999e-05 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864276+0000", 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9622000000000003e-05 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864338+0000", 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1805999999999997e-05 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.865604+0000", 2024-09-13T05:19:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656670000000001 2024-09-13T05:19:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.865675+0000", 2024-09-13T05:19:55.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0739000000000002e-05 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:77 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'192, mlcod=65'190)", 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.3508152530000004, 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020134300000000001, 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909606+0000", 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909624+0000", 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7799999999999999e-05 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909657+0000", 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3541999999999998e-05 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909676+0000", 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.861e-05 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909774+0000", 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8009000000000003e-05 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909818+0000", 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4332000000000002e-05 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911530+0000", 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712129 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911622+0000", 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1228000000000001e-05 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 430 entries 65'193 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'191)", 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.3463414220000001, 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018374859999999999, 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.332 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914080+0000", 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914105+0000", 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5726999999999999e-05 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914150+0000", 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4879000000000002e-05 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914188+0000", 2024-09-13T05:19:55.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8099999999999998e-05 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914334+0000", 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014600099999999999 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915920+0000", 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015852329999999999 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 438 entries 65'194 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:88 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'192)", 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.326961067, 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001852292, 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933460+0000", 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933481+0000", 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0746000000000001e-05 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933524+0000", 2024-09-13T05:19:55.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.278e-05 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933562+0000", 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8769000000000001e-05 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933738+0000", 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017590699999999999 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.935315+0000", 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00157636 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:149 2.3 e66/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 66'198, mlcod=65'196)", 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.9686522809999998, 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001636668, 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291770+0000", 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291783+0000", 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3030000000000001e-05 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291808+0000", 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5075999999999998e-05 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291820+0000", 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2092999999999999e-05 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291900+0000", 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0465999999999998e-05 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291958+0000", 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7346000000000003e-05 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.293350+0000", 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001392784 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.293408+0000", 2024-09-13T05:19:55.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7392999999999998e-05 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5242.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'201, mlcod=66'199)", 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.8530821040000003, 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897480000000001, 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407339+0000", 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407360+0000", 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0703000000000002e-05 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407422+0000", 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2426000000000002e-05 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407442+0000", 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407568+0000", 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126019 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407656+0000", 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7880999999999994e-05 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409045+0000", 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001389798 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409131+0000", 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5741999999999993e-05 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5248.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'204, mlcod=66'202)", 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7519411030000001, 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017322259999999999, 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508480+0000", 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508498+0000", 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7955999999999999e-05 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508532+0000", 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3886e-05 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508547+0000", 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.484e-05 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508657+0000", 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110522 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508720+0000", 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2345e-05 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510147+0000", 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014275119999999999 2024-09-13T05:19:55.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510215+0000", 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7402000000000003e-05 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:55.351 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:55.353 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:55.353 INFO:teuthology.orchestra.run.smithi016.stderr:set 3.6 pg_upmap mapping to [1,3] 2024-09-13T05:19:55.354 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:19:55.354 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:55.354 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:55.356 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:55.356 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:55.357 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:55.349+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:55.357 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:55.357 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:55.373 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:55.373 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:55.390 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:55.409 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:55.410 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:55.453 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:55.452+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:55.487 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:55.487 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:55.501 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:55.592 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:55.592+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:55.655 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:55.656+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:55.665 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:55.665 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:55.679 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:55.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:55.753+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:55.845 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:55.845 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:55.856 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:55.853+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:55.859 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:55.957 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:55.956+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:55.976 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:55.976 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:55.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:56.058 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:56.057+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:56.153 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:56.153 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:56.159 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:56.157+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:56.167 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:56.260 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:56.257+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:56.289 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:56.289 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:56.305 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:56.361 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:56.360+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:56.425 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:56.425 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:56.440 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:56.483 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:56.484+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:56.563 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:56.560+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:56.608 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:56.608 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:56.620 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:56.663 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:56.664+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:56.765 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:56.761+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:56.786 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:56.786 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:56.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:19:56.866 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:56.864+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:56.937 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:56.937 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:56.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:19:57.002 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:57.001+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:57.069 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:57.065+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:57.077 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:57.077 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:57.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:19:57.178 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:57.177+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:57.262 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:57.263 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:57.271 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:57.272+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:57.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:19:57.356 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:19:57.372 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:57.372+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:57.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:19:57.437 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:57.437 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:57.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:19:57.473 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:57.472+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:57.574 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:57.573+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:57.583 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:57.612 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:57.612 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:57.616 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:57.616 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:57.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:19:57.633 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:57.633 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:57.653 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:57.674 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:57.674 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:57.674 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:57.673+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:57.694 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.23487548999999, 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:19:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:19:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:19:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:19:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.23460346500001, 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.234559879, 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:19:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:19:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.234527652, 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:19:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.23431650400001, 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:19:57.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:19:57.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:19:57.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:19:57.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:19:57.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:19:57.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.234280211, 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:19:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:57.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:57.787 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.23424176100001, 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:19:57.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:19:57.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:19:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.234066859, 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:19:57.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:19:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:19:57.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.419362921000001, 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:57.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.411832808, 2024-09-13T05:19:57.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:19:57.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.402263480999999, 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:19:57.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:19:57.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:19:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.391100608999999, 2024-09-13T05:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:19:57.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.384941011999999, 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.820 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.825 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.336105232000001, 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:19:57.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:57.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:19:57.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:19:57.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.830 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.327828697999998, 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:19:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:19:57.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.8350682899999997, 2024-09-13T05:19:57.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:19:57.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.8346331429999996, 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:19:57.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:19:57.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:19:57.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:19:57.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.8343191509999999, 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.8339138999999998, 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:57.852 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:19:57.857 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.8335893619999997, 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:19:57.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:57.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:57.861 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:57.862 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:57.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:19:57.863 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:57.860+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:57.964 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:57.964+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:58.037 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:19:58.037 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:58.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:19:58.098 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:58.097+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:58.166 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:58.165+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:58.177 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:19:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:19:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:19:58.178 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:58.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:19:58.268 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:58.268+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:58.328 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:19:58.350 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 20% complete... Creating snap: 100% complete...done. 2024-09-13T05:19:58.359 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-13T05:19:58.359 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:58.360 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:58.361 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:58.362 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:19:58.364 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:19:58.364 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:58.364 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WvgaeitFGU/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-13T05:19:58.369 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:58.368+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:58.370 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:19:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:19:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:19:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:19:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.041362311999997, 2024-09-13T05:19:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:19:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:19:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:19:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:19:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.030156550999997, 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:19:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:19:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.019977496999999, 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:19:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.012541693999999, 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:19:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:19:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.005933990999999, 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:19:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:19:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:19:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:19:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:19:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:19:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:19:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:19:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:19:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:19:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:19:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.996707772999997, 2024-09-13T05:19:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:19:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4797.0:1 3.3 e48/47 3:c4c92e5a:::rbd_trash:head v 48'1, mlcod=0'0)", 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.909213858999998, 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015947019999999999, 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459810+0000", 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8649999999999998e-06 2024-09-13T05:19:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459820+0000", 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9899999999999992e-06 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459829+0000", 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.431e-06 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459885+0000", 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5868000000000001e-05 2024-09-13T05:19:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459912+0000", 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7203e-05 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461364+0000", 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014515820000000001 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461399+0000", 2024-09-13T05:19:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5063e-05 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.501998372999999, 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:19:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.369035514, 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:19:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:19:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4982.0:50 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:5 [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.494041701, 2024-09-13T05:19:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001569698, 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4982", 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1259763078", 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 50 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874976+0000", 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967296 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874978+0000", 2024-09-13T05:19:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2100000000000001e-06 2024-09-13T05:19:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874981+0000", 2024-09-13T05:19:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1760000000000002e-06 2024-09-13T05:19:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874984+0000", 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9569999999999998e-06 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876017+0000", 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00103304 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876260+0000", 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000243403 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876546+0000", 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028614500000000002 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.39047476, 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017381950000000001, 2024-09-13T05:19:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978546+0000", 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2059999999999999e-06 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978555+0000", 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5969999999999992e-06 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978561+0000", 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7390000000000002e-06 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979976+0000", 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014147669999999999 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980108+0000", 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013226399999999999 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980282+0000", 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017327799999999999 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'182, mlcod=65'180)", 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.6794866860000006, 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016537559999999999, 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689530+0000", 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689546+0000", 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6439999999999998e-05 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689575+0000", 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9159999999999999e-05 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689590+0000", 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5056e-05 2024-09-13T05:19:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689724+0000", 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133518 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689793+0000", 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8769000000000005e-05 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691110+0000", 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001317807 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691185+0000", 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4876000000000006e-05 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5233.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'183, mlcod=65'181)", 2024-09-13T05:19:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.5116769649999995, 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001624459, 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:19:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857339+0000", 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857354+0000", 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5157000000000001e-05 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857383+0000", 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8759e-05 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857396+0000", 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3223e-05 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857503+0000", 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010699199999999999 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857575+0000", 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1092000000000003e-05 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.858894+0000", 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001319408 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.858966+0000", 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1811999999999996e-05 2024-09-13T05:19:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5233.0:16 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'185, mlcod=65'183)", 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.864111+0000", 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.5049076590000006, 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001563947, 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864111+0000", 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864111+0000", 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864109+0000", 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864125+0000", 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.624e-05 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864159+0000", 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3846000000000001e-05 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864177+0000", 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7629999999999999e-05 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864276+0000", 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9622000000000003e-05 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864338+0000", 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1805999999999997e-05 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.865604+0000", 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656670000000001 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.865675+0000", 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0739000000000002e-05 2024-09-13T05:19:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:77 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'192, mlcod=65'190)", 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.4594099949999997, 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020134300000000001, 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909606+0000", 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909624+0000", 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7799999999999999e-05 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909657+0000", 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3541999999999998e-05 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909676+0000", 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.861e-05 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909774+0000", 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8009000000000003e-05 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909818+0000", 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4332000000000002e-05 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911530+0000", 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712129 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911622+0000", 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1228000000000001e-05 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 430 entries 65'193 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'191)", 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.4549361639999994, 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018374859999999999, 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914080+0000", 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914105+0000", 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5726999999999999e-05 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914150+0000", 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4879000000000002e-05 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914188+0000", 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8099999999999998e-05 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914334+0000", 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014600099999999999 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915920+0000", 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015852329999999999 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 438 entries 65'194 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:88 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'192)", 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:19:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.4355558090000002, 2024-09-13T05:19:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001852292, 2024-09-13T05:19:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:19:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:19:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933460+0000", 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933481+0000", 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0746000000000001e-05 2024-09-13T05:19:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933524+0000", 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.278e-05 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933562+0000", 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8769000000000001e-05 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933738+0000", 2024-09-13T05:19:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017590699999999999 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.935315+0000", 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00157636 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:149 2.3 e66/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 66'198, mlcod=65'196)", 2024-09-13T05:19:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.077247023, 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001636668, 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291770+0000", 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291783+0000", 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3030000000000001e-05 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291808+0000", 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5075999999999998e-05 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291820+0000", 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2092999999999999e-05 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291900+0000", 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0465999999999998e-05 2024-09-13T05:19:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291958+0000", 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7346000000000003e-05 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.293350+0000", 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001392784 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.293408+0000", 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7392999999999998e-05 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5242.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'201, mlcod=66'199)", 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.9616768459999996, 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897480000000001, 2024-09-13T05:19:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407339+0000", 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407360+0000", 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0703000000000002e-05 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407422+0000", 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2426000000000002e-05 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407442+0000", 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407568+0000", 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126019 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407656+0000", 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7880999999999994e-05 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409045+0000", 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001389798 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409131+0000", 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5741999999999993e-05 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5248.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'204, mlcod=66'202)", 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.8605358450000002, 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017322259999999999, 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508480+0000", 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508498+0000", 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7955999999999999e-05 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508532+0000", 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3886e-05 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508547+0000", 2024-09-13T05:19:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.484e-05 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508657+0000", 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110522 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508720+0000", 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2345e-05 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510147+0000", 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014275119999999999 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510215+0000", 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7402000000000003e-05 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:19:58.469 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:19:58.472 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139281 s, 75.3 MB/s 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 23556 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=23556 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:19:58.473 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:19:58.474 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:19:58.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:19:58.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:19:58.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:19:58.474 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:58.468+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:58.572 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:58.569+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:58.604 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:58.605 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:58.605 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:58.605 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:58.618 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:19:58.672 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:58.672+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:58.734 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:58.735 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:58.735 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:58.735 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:58.735 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:58.749 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:19:58.773 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:58.769+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:58.874 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:58.872+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:58.918 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:19:58.918 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.783960913, 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:19:58.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.783329654, 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:58.927 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.78268453299999, 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:19:58.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:19:58.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:19:58.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.931 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.78217717, 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:19:58.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:19:58.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:19:58.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:19:58.935 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.78165173399999, 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:19:58.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:19:58.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:19:58.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:19:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:19:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.781014222, 2024-09-13T05:19:58.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:19:58.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:19:58.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:19:58.944 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.780437514, 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.945 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.948 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.779886041, 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.949 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:19:58.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.77942545800001, 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.778915572, 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:19:58.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:19:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:19:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:19:58.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:19:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:19:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:19:58.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.962 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.778488509, 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:19:58.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:19:58.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:19:58.966 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 108.39172879100001, 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 107.774193979, 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.971 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:19:58.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.975 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.543415044, 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:58.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:19:58.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:58.980 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.532794699999997, 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:58.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:19:58.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:58.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.525580386000001, 2024-09-13T05:19:58.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.991 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.519224866000002, 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:19:58.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:19:58.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:19:58.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:58.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:19:58.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.997 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.519168501999999, 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:58.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:58.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:19:59.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:19:59.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.003 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.519130930999999, 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:59.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:19:59.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:59.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.519096406999999, 2024-09-13T05:19:59.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:19:59.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:59.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:19:59.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:19:59.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.015 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.015 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.015 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:59.017 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:19:59.017 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:59.016+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:59.118 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:19:59.116+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:19:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:59.186 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:59.186 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:59.200 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:19:59.218 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:19:59.217+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:19:59.319 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:19:59.317+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:19:59.360 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:59.360 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:59.360 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:19:59.360 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:19:59.360 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:59.374 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:19:59.420 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:19:59.420+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.151355721000002, 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.494 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:19:59.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:19:59.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:59.500 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.139475118999997, 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:19:59.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:19:59.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.133531419999997, 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.505 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:19:59.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.130855711999999, 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:19:59.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:19:59.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.126141679, 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:19:59.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:19:59.518 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.110189406000003, 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:19:59.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:19:59.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:19:59.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.100549204000004, 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:59.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.093456398999997, 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:19:59.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:19:59.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.085238443000001, 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:59.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:19:59.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:59.536 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.083092630000003, 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:19:59.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.540 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.081174836000002, 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:59.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:19:59.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:19:59.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.927728574, 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.546 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:19:59.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:19:59.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:19:59.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:19:59.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.921290634000002, 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.557 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.915222481000001, 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.558 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:59.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.907678927999999, 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.562 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:19:59.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:19:59.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:19:59.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:19:59.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.566 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.566 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:19:59.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:19:59.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.567 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 31.895001683, 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:19:59.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:19:59.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:19:59.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:19:59.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:19:59.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:19:59.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:19:59.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 23.408377817000002, 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.573 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:19:59.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:19:59.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 23.408308995999999, 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:19:59.578 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:19:59.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:19:59.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:19:59.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:19:59.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:19:59.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:19:59.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.583 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:19:59.584 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:19:59.584 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 19.379906426000002, 2024-09-13T05:19:59.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:19:59.584 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.584 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.584 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 10.853226112, 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:19:59.591 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:19:59.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:19:59.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:19:59.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:19:59.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:19:59.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:19:59.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:19:59.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:19:59.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:19:59.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:19:59.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:19:59.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:19:59.598 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:59.600 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:19:59.600 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:59.597+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:59.701 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:59.700+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:59.714 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:19:59.768 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:19:59.768 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:19:59.768 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:19:59.768 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:19:59.782 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:19:59.801 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:19:59.801+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:19:59.872 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:19:59.872 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:19:59.888 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:19:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:19:59.902 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:19:59.900+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:19:59.904 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:19:59.904 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:19:59.924 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:19:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:19:59.944 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:19:59.944 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:19:59.947 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:19:59.947 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:19:59.962 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:19:59.963 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:19:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:00.003 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:00.004+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:00.103 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:00.104+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:00.117 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:00.117 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:00.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:00.204 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:00.201+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:00.264 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:00.264 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:00.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:00.305 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:00.304+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:00.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:20:00.356 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:20:00.356 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:20:00.357 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T05:20:00.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T05:20:00.405 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:00.404+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:00.443 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:00.443 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:00.459 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:00.546 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:00.545+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:00.608 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:00.608+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:00.620 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:00.620 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:00.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:00.657 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 70, 2024-09-13T05:20:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5", 2024-09-13T05:20:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2024-09-13T05:17:57.363809+0000", 2024-09-13T05:20:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2024-09-13T05:19:58.317866+0000", 2024-09-13T05:20:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2024-09-13T05:19:34.047702+0000", 2024-09-13T05:20:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2024-09-13T05:17:59.738330+0000", 2024-09-13T05:20:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5799936, 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 7, 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T05:20:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 3, 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "reef", 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_crimson": false, 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": ".mgr", 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:08.122164+0000", 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2024-09-13T05:20:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2024-09-13T05:20:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "15", 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:20:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:20:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:20:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": {} 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 6.059999942779541, 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 6.059999942779541, 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.33000001311302185, 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2, 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2, 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:15.001664+0000", 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:20:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:20:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "70", 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 7, 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 70, 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:20:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:20:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:20:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 2.25, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 2.25, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 1, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2.25, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2.25, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:20:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 3, 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "new_default_pool21435", 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:19:21.793970+0000", 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:20:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:20:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "50", 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:20:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 3, 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 50, 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:20:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:20:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:20:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 2.25, 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 2.25, 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 1, 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2.25, 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2.25, 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99680078029632568, 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "e58174f4-85ec-4321-8bc3-384261c59c6b", 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:20:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 48, 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-13T05:20:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-13T05:20:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/2153409514", 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/2153409514", 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/2153409514", 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/2153409514", 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "84a3db6c-dacd-43d5-b964-37fdef0091a4", 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:20:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:20:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:20:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 68, 2024-09-13T05:20:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:20:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2024-09-13T05:20:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2024-09-13T05:20:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/2608485151", 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/2608485151", 2024-09-13T05:20:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/2608485151", 2024-09-13T05:20:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/2608485151", 2024-09-13T05:20:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "0d530476-cf38-4544-9ce7-e8ed35df220b", 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 47, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2024-09-13T05:20:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/1006404377", 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/1006404377", 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/1006404377", 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/1006404377", 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "fd91f7e2-580d-47ea-9db5-792c8f1ebe01", 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.980804443359375, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 20, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 29, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 47, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 21, 2024-09-13T05:20:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6816", 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6817", 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6818", 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6819", 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6822", 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6823", 2024-09-13T05:20:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6820", 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6821", 2024-09-13T05:20:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6817/1834661236", 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6819/1834661236", 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6823/1834661236", 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6821/1834661236", 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "01af0f4c-5da2-46fe-a262-59fcc6a2efa9", 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 38, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 54, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 54, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 40, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6808", 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6809", 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6810", 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6811", 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6814", 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6815", 2024-09-13T05:20:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6812", 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6813", 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6809/3457694380", 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6811/3457694380", 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6815/3457694380", 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6813/3457694380", 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "64cdd5f2-1568-4e96-ad90-6ef7414651f0", 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 47, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:20:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6800", 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6801", 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6802", 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6803", 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6806", 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6807", 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6804", 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6805", 2024-09-13T05:20:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6801/2778785304", 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6803/2778785304", 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6807/2778785304", 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6805/2778785304", 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.193869+0000", 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.255764+0000", 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.286354+0000", 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.221310+0000", 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 21 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.026235+0000", 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 40 2024-09-13T05:20:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.010242+0000", 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2024-09-13T05:20:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: 0, 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "3.6", 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: 3 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_primaries": [], 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2024-09-13T05:20:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2024-09-13T05:20:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2024-09-13T05:20:00.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:00.726 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:00.728 INFO:tasks.thrashosds.thrasher:j is {'epoch': 70, 'fsid': '1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5', 'created': '2024-09-13T05:17:57.363809+0000', 'modified': '2024-09-13T05:19:58.317866+0000', 'last_up_change': '2024-09-13T05:19:34.047702+0000', 'last_in_change': '2024-09-13T05:17:59.738330+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T05:18:08.122164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T05:18:15.001664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool21435', 'create_time': '2024-09-13T05:19:21.793970+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 50, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9968007802963257, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e58174f4-85ec-4321-8bc3-384261c59c6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 2153409514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 2153409514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 2153409514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 2153409514}]}, 'public_addr': '172.21.15.16:6801/2153409514', 'cluster_addr': '172.21.15.16:6803/2153409514', 'heartbeat_back_addr': '172.21.15.16:6807/2153409514', 'heartbeat_front_addr': '172.21.15.16:6805/2153409514', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '84a3db6c-dacd-43d5-b964-37fdef0091a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 68, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 2608485151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 2608485151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 2608485151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 2608485151}]}, 'public_addr': '172.21.15.16:6809/2608485151', 'cluster_addr': '172.21.15.16:6811/2608485151', 'heartbeat_back_addr': '172.21.15.16:6815/2608485151', 'heartbeat_front_addr': '172.21.15.16:6813/2608485151', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0d530476-cf38-4544-9ce7-e8ed35df220b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 1006404377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 1006404377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 1006404377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 1006404377}]}, 'public_addr': '172.21.15.16:6817/1006404377', 'cluster_addr': '172.21.15.16:6819/1006404377', 'heartbeat_back_addr': '172.21.15.16:6823/1006404377', 'heartbeat_front_addr': '172.21.15.16:6821/1006404377', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fd91f7e2-580d-47ea-9db5-792c8f1ebe01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.980804443359375, 'last_clean_begin': 12, 'last_clean_end': 20, 'up_from': 29, 'up_thru': 47, 'down_at': 21, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6816', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6817', 'nonce': 1834661236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6818', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6819', 'nonce': 1834661236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6822', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6823', 'nonce': 1834661236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6820', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6821', 'nonce': 1834661236}]}, 'public_addr': '172.21.15.79:6817/1834661236', 'cluster_addr': '172.21.15.79:6819/1834661236', 'heartbeat_back_addr': '172.21.15.79:6823/1834661236', 'heartbeat_front_addr': '172.21.15.79:6821/1834661236', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '01af0f4c-5da2-46fe-a262-59fcc6a2efa9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 12, 'last_clean_end': 38, 'up_from': 54, 'up_thru': 54, 'down_at': 40, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6808', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6809', 'nonce': 3457694380}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6810', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6811', 'nonce': 3457694380}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6814', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6815', 'nonce': 3457694380}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6812', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6813', 'nonce': 3457694380}]}, 'public_addr': '172.21.15.79:6809/3457694380', 'cluster_addr': '172.21.15.79:6811/3457694380', 'heartbeat_back_addr': '172.21.15.79:6815/3457694380', 'heartbeat_front_addr': '172.21.15.79:6813/3457694380', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '64cdd5f2-1568-4e96-ad90-6ef7414651f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6800', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6801', 'nonce': 2778785304}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6802', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6803', 'nonce': 2778785304}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6806', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6807', 'nonce': 2778785304}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6804', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6805', 'nonce': 2778785304}]}, 'public_addr': '172.21.15.79:6801/2778785304', 'cluster_addr': '172.21.15.79:6803/2778785304', 'heartbeat_back_addr': '172.21.15.79:6807/2778785304', 'heartbeat_front_addr': '172.21.15.79:6805/2778785304', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.193869+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.255764+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.286354+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.221310+0000', 'dead_epoch': 21}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.026235+0000', 'dead_epoch': 40}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.010242+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [0, 1]}, {'pgid': '3.6', 'osds': [1, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T05:20:00.728 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-13T05:20:00.729 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:00.728+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:00.828 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:00.825+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:00.851 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:00.851 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:00.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:00.950 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:00.949+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:01.016 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:01.017 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:01.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:01.031 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:01.029+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:01.131 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:01.128+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:01.193 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:01.193 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:01.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:01.232 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:01.232+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:01.314 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:01.314 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:01.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:01.333 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:01.332+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:20:01.434 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:01.432+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:01.439 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:01.439 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:01.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:01.535 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:01.532+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:01.574 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:01.575 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:01.589 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:01.636 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:01.636+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:01.722 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:01.722 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:01.736 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:01.737 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:01.736+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:01.837 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:01.836+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:01.900 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:01.900 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:01.914 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:01.956 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:01.956+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:01.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:02.039 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:02.040+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:02.076 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:02.076 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:02.091 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:02.140 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:02.140 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:02.140 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:02.137+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:02.156 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:02.171 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:02.171 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:02.189 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:02.206 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:02.206 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:02.225 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:02.225 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:02.242 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:02.240+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:02.261 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:02.261 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:02.261 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:02.261 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:02.275 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:02.363 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:02.364+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:02.441 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:02.442 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:02.442 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:02.442 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:02.442 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:02.443 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:02.441+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:02.455 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:02.544 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:02.541+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:02.630 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:02.630 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.287782190000001, 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:20:02.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.632 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:20:02.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:20:02.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:02.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.636 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.275901587999996, 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:20:02.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:20:02.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:20:02.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.269957888999997, 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:20:02.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.267282180999999, 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:02.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:20:02.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:20:02.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.262568148, 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.649 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:20:02.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:20:02.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:20:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:02.653 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.246615875000003, 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:20:02.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.657 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.236975673000003, 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:02.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:20:02.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:20:02.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.661 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.229882867999997, 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.662 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:20:02.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:20:02.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.221664912000001, 2024-09-13T05:20:02.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:20:02.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.219519099000003, 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.672 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:20:02.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:20:02.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.217601305000002, 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:02.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:20:02.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.678 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:02.679 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 35.064155043, 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:02.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:20:02.683 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 35.057717103000002, 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:02.684 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:20:02.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:20:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 35.051648950000001, 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:02.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:20:02.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.691 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:20:02.692 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 35.044105397000003, 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:02.693 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.695 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:20:02.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 35.031428151999997, 2024-09-13T05:20:02.697 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:20:02.700 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.544804286000002, 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.702 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:20:02.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.705 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:02.706 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 26.544735464999999, 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:02.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 22.516332895000001, 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:20:02.711 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:20:02.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:02.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.716 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:20:02.717 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 13.989652581, 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:02.718 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:02.719 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.720 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:20:02.721 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:02.723 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:02.725 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:02.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:02.721+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:02.826 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:02.824+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:02.849 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:02.849 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:02.849 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:02.849 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:02.865 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:02.955 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:02.956+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:03.028 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:03.025+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:03.030 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:03.030 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:03.044 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:03.129 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:03.125+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:03.208 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:03.208 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:03.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:03.208 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:03.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.208 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.160908535000001, 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.210 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:03.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:03.213 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.124507956999999, 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:03.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:03.217 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.120924652999999, 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:03.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.116290567, 2024-09-13T05:20:03.221 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.222 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.113053291, 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:03.225 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.226 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.227 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:03.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.109373892000001, 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:03.229 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.231 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.232 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.106017698999999, 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.233 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.234 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.235 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.236 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.099686406, 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.237 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:03.239 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.240 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.092006457, 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.241 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:03.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.244 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.089375313000001, 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.245 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.248 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.085533352999999, 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.249 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:03.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.252 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.081697956999999, 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.253 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.078242711000001, 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.257 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:03.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.074168871000001, 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.261 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.262 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.263 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.071089942, 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:03.268 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.066846926, 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.269 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:03.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.272 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.062970923999998, 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.273 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:03.274 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.275 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.055345298999999, 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.277 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:03.278 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.279 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.280 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.052078889000001, 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.281 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.282 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:03.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.284 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:03.285 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 28.048386485999998, 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:03.286 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:03.287 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.288 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:03.289 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:03.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:03.291 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:03.289+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:03.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:20:03.406 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:03.405+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:03.426 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:03.426 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:03.426 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:03.426 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:03.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:03.492 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:03.492+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:03.593 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:03.592+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:03.602 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:03.602 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:03.602 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:03.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:03.694 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:03.693+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.607112453, 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:20:03.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:20:03.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 105.597330923, 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:20:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:20:03.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.374500670000003, 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.756 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:20:03.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.367471074999997, 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:03.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:03.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.765 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.326688240000003, 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.766 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:20:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:20:03.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.319829767000002, 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:20:03.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.287687806000001, 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:20:03.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:20:03.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:20:03.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:20:03.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.673726332000001, 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:20:03.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.673565106000002, 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:20:03.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.793 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:80 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'83, mlcod=57'80)", 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.824828048000001, 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072315670000000004, 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915606+0000", 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915894+0000", 2024-09-13T05:20:03.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028823899999999998 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915934+0000", 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9659000000000001e-05 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915953+0000", 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8660000000000001e-05 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916091+0000", 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790800000000001 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916150+0000", 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9079000000000001e-05 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922698+0000", 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065481339999999997 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922839+0000", 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140805 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:81 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'84, mlcod=57'80)", 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.819617554000001, 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003098127, 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920815+0000", 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920950+0000", 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502499999999999 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920989+0000", 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8779000000000003e-05 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921013+0000", 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3753000000000001e-05 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921422+0000", 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040899799999999999 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921481+0000", 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8811999999999999e-05 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923815+0000", 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002333852 2024-09-13T05:20:03.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923916+0000", 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101032 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:82 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'85, mlcod=57'80)", 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.922538+0000", 2024-09-13T05:20:03.803 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.817897147, 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002777046, 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922538+0000", 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922538+0000", 2024-09-13T05:20:03.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922536+0000", 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922652+0000", 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116105 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922682+0000", 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9975999999999999e-05 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922701+0000", 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8159999999999999e-05 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922927+0000", 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022668300000000001 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922996+0000", 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8619000000000001e-05 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.925244+0000", 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022477859999999999 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.925315+0000", 2024-09-13T05:20:03.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1313e-05 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:159 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.302269+0000", 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.438165783000001, 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026821850000000001, 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:03.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 159 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302269+0000", 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302269+0000", 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302265+0000", 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302287+0000", 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1353999999999999e-05 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302335+0000", 2024-09-13T05:20:03.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7715e-05 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302367+0000", 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1977000000000002e-05 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302514+0000", 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147424 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:03.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302534+0000", 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9760000000000001e-05 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.304769+0000", 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022353669999999998 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.304849+0000", 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9797999999999997e-05 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.304951+0000", 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010253 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:169 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'92, mlcod=65'89)", 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.410008969, 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004357662, 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330424+0000", 2024-09-13T05:20:03.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330542+0000", 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117475 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330570+0000", 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8036999999999999e-05 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330585+0000", 2024-09-13T05:20:03.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4983e-05 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330651+0000", 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6361999999999999e-05 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330674+0000", 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3064000000000001e-05 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334660+0000", 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039862229999999997 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334784+0000", 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012329499999999999 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:170 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'93, mlcod=65'89)", 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.404812321, 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035268080000000002, 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335621+0000", 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335688+0000", 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7006000000000002e-05 2024-09-13T05:20:03.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335716+0000", 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.798e-05 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335728+0000", 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1712999999999999e-05 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335838+0000", 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110575 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335878+0000", 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9969000000000003e-05 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339094+0000", 2024-09-13T05:20:03.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032155399999999998 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339149+0000", 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5685e-05 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:172 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'95, mlcod=65'89)", 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.402671718000001, 2024-09-13T05:20:03.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029153970000000001, 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337762+0000", 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337854+0000", 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2342999999999993e-05 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337868+0000", 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3658e-05 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337876+0000", 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5900000000000002e-06 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338010+0000", 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013445199999999999 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338063+0000", 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2762000000000001e-05 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340641+0000", 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002577663 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340679+0000", 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8172e-05 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:174 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'97, mlcod=65'90)", 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.338917+0000", 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.401517689, 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027662759999999998, 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338917+0000", 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338917+0000", 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338915+0000", 2024-09-13T05:20:03.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339160+0000", 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024431000000000002 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339190+0000", 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0171999999999998e-05 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339208+0000", 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8053000000000001e-05 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339628+0000", 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042042700000000002 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339691+0000", 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2212999999999995e-05 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.341629+0000", 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019388770000000001 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.341684+0000", 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4144000000000002e-05 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.831 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:238 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'101, mlcod=66'99)", 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.2913417, 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048419240000000001, 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449091+0000", 2024-09-13T05:20:03.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450611+0000", 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015204629999999999 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450655+0000", 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3662999999999999e-05 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450676+0000", 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0573000000000001e-05 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450825+0000", 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014922399999999999 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450859+0000", 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3779999999999998e-05 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453864+0000", 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030051959999999999 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453935+0000", 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1302000000000003e-05 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:323 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'103, mlcod=66'100)", 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.3631914570000001, 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029606749999999999, 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:03.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377243+0000", 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377314+0000", 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1248999999999995e-05 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377333+0000", 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924999999999999e-05 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377344+0000", 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0817e-05 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377395+0000", 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0803000000000001e-05 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377416+0000", 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0993e-05 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380171+0000", 2024-09-13T05:20:03.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027549660000000002 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380204+0000", 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3331999999999999e-05 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:324 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'104, mlcod=66'101)", 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.382617+0000", 2024-09-13T05:20:03.841 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.3578181330000003, 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002752174, 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.382617+0000", 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.382617+0000", 2024-09-13T05:20:03.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.382615+0000", 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383431+0000", 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081575199999999999 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:03.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383459+0000", 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7650000000000001e-05 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383471+0000", 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2578000000000001e-05 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383520+0000", 2024-09-13T05:20:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9311000000000003e-05 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383538+0000", 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7367000000000002e-05 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.385332+0000", 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001794192 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:03.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.385369+0000", 2024-09-13T05:20:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7064999999999998e-05 2024-09-13T05:20:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:03.846 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:03.846 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:03.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:03.848 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:03.845+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:03.948 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:03.945+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:04.019 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:04.020 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:04.020 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:04.020 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:04.034 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:04.049 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:04.048+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:04.150 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:04.149+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:04.206 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:04.206 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:04.206 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:04.206 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:04.206 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:04.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:04.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:04.266 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:04.265+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:04.351 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:04.352+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:04.359 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:20:04.398 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:04.398 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:04.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 113.881529808, 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:20:04.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:20:04.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:20:04.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 113.881257783, 2024-09-13T05:20:04.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:20:04.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:04.411 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 113.88121419700001, 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:20:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:04.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.417 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 113.88118197, 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:20:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.423 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 113.88097082199999, 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:04.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:04.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:20:04.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:04.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:20:04.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:20:04.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:20:04.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:20:04.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:20:04.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:20:04.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:20:04.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:20:04.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:20:04.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:20:04.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:04.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:20:04.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:20:04.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:20:04.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:20:04.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:20:04.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:20:04.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:20:04.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:20:04.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:20:04.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:20:04.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.434 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.434 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:04.434 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 113.880934529, 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:20:04.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:04.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 113.880896079, 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.441 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:20:04.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:20:04.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:20:04.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 113.880721177, 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:04.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:20:04.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:20:04.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:20:04.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.066017238999997, 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.453 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:04.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:20:04.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:20:04.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:04.459 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.058487126000003, 2024-09-13T05:20:04.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:20:04.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:04.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:04.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:20:04.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.461 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:04.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:04.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:20:04.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:04.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:20:04.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:20:04.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:20:04.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:20:04.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:20:04.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:20:04.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:20:04.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:20:04.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:20:04.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:20:04.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:20:04.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:20:04.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:20:04.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:20:04.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:20:04.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:20:04.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.467 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.468 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:04.468 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:04.468 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.048917799000002, 2024-09-13T05:20:04.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:20:04.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.469 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:04.469 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:04.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:20:04.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.469 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:04.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:04.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:20:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:20:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:20:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:20:04.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:20:04.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:20:04.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:20:04.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:20:04.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:20:04.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:20:04.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.037754927000002, 2024-09-13T05:20:04.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:20:04.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:20:04.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:20:04.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.031595330000002, 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:20:04.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:20:04.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.982759549999997, 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.487 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:20:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:04.492 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.974483016000001, 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:04.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:20:04.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.481722608, 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:04.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:20:04.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.481287461000001, 2024-09-13T05:20:04.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:04.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.510 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.480973469, 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:04.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.516 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.480568218, 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:20:04.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:04.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:04.522 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.480243679999999, 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:20:04.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:04.526 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:04.527 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:04.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:04.529 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:04.529 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:04.530 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:04.530 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:04.531 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:04.531 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:04.532 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:04.532 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:04.528+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:04.629 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:04.628+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:04.694 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:04.695 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:04.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:04.730 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:04.728+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:04.831 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:04.828+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:04.836 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:04.837 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:04.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:04.932 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:04.929+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:04.970 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:04.970 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:04.984 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:05.033 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:05.032+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:05.114 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:05.114 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:05.128 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:05.171 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:05.172+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:05.234 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:05.232+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:05.292 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:05.292 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:05.306 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:05.335 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:05.333+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:05.374 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:20:05.374 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23556\b' 2024-09-13T05:20:05.374 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23556\b' 2024-09-13T05:20:05.432 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:05.433 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:20:05.433 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *23556\b' 2024-09-13T05:20:05.433 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *23556\b' 2024-09-13T05:20:05.435 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:05.436+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:05.439 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:05.439 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:05.469 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:05.469 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:05.483 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:05.571 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:05.572+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:05.637 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:05.633+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:05.651 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:05.651 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:05.666 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:05.728 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:20:05.729 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:20:05.729 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-13T05:20:05.729 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T05:20:05.729 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T05:20:05.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-13T05:20:05.751 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:05.752+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:05.831 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:05.831 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:05.839 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:05.837+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:05.847 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:05.864 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:05.864 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T05:20:05.864 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:05.939 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:05.940+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:05.941 INFO:tasks.workunit.client.0.smithi122.stderr:+ LOG_FILE=/tmp/tmp.WvgaeitFGU/rbd-nbd.log 2024-09-13T05:20:05.941 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n '' ']' 2024-09-13T05:20:05.941 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map --quiesce rbd/testrbdnbd21435 --log-file=/tmp/tmp.WvgaeitFGU/rbd-nbd.log 2024-09-13T05:20:05.941 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:20:05.942 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:20:05.943 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:20:05.944 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:20:05.946 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:20:05.946 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:20:05.947 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce rbd/testrbdnbd21435 --log-file=/tmp/tmp.WvgaeitFGU/rbd-nbd.log 2024-09-13T05:20:06.015 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:06.015 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:06.029 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:06.072 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:06.072+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:06.104 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd0 2024-09-13T05:20:06.104 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:06.104 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:06.104 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:06.104 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:06.105 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:06.142 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:06.141+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:06.169 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23738 2024-09-13T05:20:06.169 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23738 2024-09-13T05:20:06.169 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23738 -C rbd-nbd 2024-09-13T05:20:06.175 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:06.175 INFO:tasks.workunit.client.0.smithi122.stdout: 23738 ? 00:00:00 rbd-nbd 2024-09-13T05:20:06.175 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo mkfs /dev/nbd0 2024-09-13T05:20:06.176 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:06.176 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:06.177 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:06.177 INFO:tasks.workunit.client.0.smithi122.stderr:++ which mkfs 2024-09-13T05:20:06.178 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/mkfs 2024-09-13T05:20:06.179 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:06.179 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/mkfs /dev/nbd0 2024-09-13T05:20:06.193 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:06.193 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:06.207 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:06.222 INFO:tasks.workunit.client.0.smithi122.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-13T05:20:06.251 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:06.252+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:06.304 INFO:tasks.workunit.client.0.smithi122.stdout:Discarding device blocks: 0/16384 done 2024-09-13T05:20:06.304 INFO:tasks.workunit.client.0.smithi122.stdout:Creating filesystem with 16384 4k blocks and 16384 inodes 2024-09-13T05:20:06.304 INFO:tasks.workunit.client.0.smithi122.stdout: 2024-09-13T05:20:06.304 INFO:tasks.workunit.client.0.smithi122.stdout:Allocating group tables: 0/1 done 2024-09-13T05:20:06.329 INFO:tasks.workunit.client.0.smithi122.stdout:Writing inode tables: 0/1 done 2024-09-13T05:20:06.336 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:06.337 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:06.343 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:06.340+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:06.351 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:06.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:20:06.444 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:06.441+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:06.467 INFO:tasks.workunit.client.0.smithi122.stdout:Writing superblocks and filesystem accounting information: 0/1 done 2024-09-13T05:20:06.467 INFO:tasks.workunit.client.0.smithi122.stdout: 2024-09-13T05:20:06.469 INFO:tasks.workunit.client.0.smithi122.stderr:+ mkdir /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:06.471 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:06.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:06.472 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:06.473 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:06.473 INFO:tasks.workunit.client.0.smithi122.stderr:++ which mount 2024-09-13T05:20:06.476 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/mount 2024-09-13T05:20:06.476 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:06.476 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:06.509 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd snap create rbd/testrbdnbd21435@quiesce3 2024-09-13T05:20:06.521 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:06.521 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:06.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:06.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:06.582 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:06.581+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:06.645 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:06.641+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:06.656 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:06.656 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:06.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:06.672 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:06.688 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:06.705 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:06.725 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:06.746 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:06.746 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:06.744+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:06.766 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:06.787 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:06.788 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:06.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:06.847 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:06.844+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:06.948 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:06.948+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:06.977 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:06.977 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:06.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:06.977 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:06.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.977 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:20:06.977 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.840495637, 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:06.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:20:06.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:20:06.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:06.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:06.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 108.83071410700001, 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:20:06.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:20:06.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:20:06.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.607883854000001, 2024-09-13T05:20:06.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:06.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:20:06.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:06.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.600854259000002, 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.995 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:20:06.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:06.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:20:06.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:20:06.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:06.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:20:06.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:20:06.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:06.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.560071424, 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.000 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:20:07.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.553212950999999, 2024-09-13T05:20:07.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:20:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.521070989999998, 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.010 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:07.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.907109515999998, 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:20:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:20:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.906948289999999, 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:20:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:80 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'83, mlcod=57'80)", 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.058211232000001, 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072315670000000004, 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915606+0000", 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915894+0000", 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028823899999999998 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915934+0000", 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9659000000000001e-05 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915953+0000", 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8660000000000001e-05 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916091+0000", 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790800000000001 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916150+0000", 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9079000000000001e-05 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922698+0000", 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065481339999999997 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922839+0000", 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140805 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:81 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'84, mlcod=57'80)", 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.053000738000001, 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003098127, 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920815+0000", 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920950+0000", 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502499999999999 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920989+0000", 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8779000000000003e-05 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921013+0000", 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3753000000000001e-05 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921422+0000", 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040899799999999999 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921481+0000", 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8811999999999999e-05 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923815+0000", 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002333852 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923916+0000", 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101032 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:82 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'85, mlcod=57'80)", 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.922538+0000", 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.051280331000001, 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002777046, 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922538+0000", 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922538+0000", 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922536+0000", 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922652+0000", 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116105 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922682+0000", 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9975999999999999e-05 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922701+0000", 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8159999999999999e-05 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922927+0000", 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022668300000000001 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922996+0000", 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8619000000000001e-05 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.925244+0000", 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022477859999999999 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.925315+0000", 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1313e-05 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:159 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.302269+0000", 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.671548967, 2024-09-13T05:20:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026821850000000001, 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 159 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302269+0000", 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302269+0000", 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302265+0000", 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302287+0000", 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1353999999999999e-05 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302335+0000", 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7715e-05 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302367+0000", 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1977000000000002e-05 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302514+0000", 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147424 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.302534+0000", 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9760000000000001e-05 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.304769+0000", 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022353669999999998 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.304849+0000", 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9797999999999997e-05 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.304951+0000", 2024-09-13T05:20:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010253 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:169 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'92, mlcod=65'89)", 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.643392153000001, 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004357662, 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330424+0000", 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330542+0000", 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117475 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330570+0000", 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8036999999999999e-05 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330585+0000", 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4983e-05 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330651+0000", 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6361999999999999e-05 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330674+0000", 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3064000000000001e-05 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334660+0000", 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039862229999999997 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334784+0000", 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012329499999999999 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:170 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'93, mlcod=65'89)", 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.638195504999999, 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035268080000000002, 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335621+0000", 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335688+0000", 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7006000000000002e-05 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335716+0000", 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.798e-05 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335728+0000", 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1712999999999999e-05 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335838+0000", 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110575 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335878+0000", 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9969000000000003e-05 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339094+0000", 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032155399999999998 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339149+0000", 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5685e-05 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:172 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'95, mlcod=65'89)", 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.636054902000001, 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029153970000000001, 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337762+0000", 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337854+0000", 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2342999999999993e-05 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337868+0000", 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3658e-05 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337876+0000", 2024-09-13T05:20:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5900000000000002e-06 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338010+0000", 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013445199999999999 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338063+0000", 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2762000000000001e-05 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340641+0000", 2024-09-13T05:20:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002577663 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340679+0000", 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8172e-05 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:174 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'97, mlcod=65'90)", 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.338917+0000", 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.634900872999999, 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027662759999999998, 2024-09-13T05:20:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338917+0000", 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338917+0000", 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338915+0000", 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339160+0000", 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024431000000000002 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339190+0000", 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0171999999999998e-05 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339208+0000", 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8053000000000001e-05 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339628+0000", 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042042700000000002 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339691+0000", 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2212999999999995e-05 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.341629+0000", 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019388770000000001 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.341684+0000", 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4144000000000002e-05 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:238 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'101, mlcod=66'99)", 2024-09-13T05:20:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.524724884000001, 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048419240000000001, 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449091+0000", 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450611+0000", 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015204629999999999 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450655+0000", 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3662999999999999e-05 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450676+0000", 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0573000000000001e-05 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450825+0000", 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014922399999999999 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450859+0000", 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3779999999999998e-05 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453864+0000", 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030051959999999999 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453935+0000", 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1302000000000003e-05 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:323 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'103, mlcod=66'100)", 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.5965746410000001, 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029606749999999999, 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377243+0000", 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377314+0000", 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1248999999999995e-05 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377333+0000", 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924999999999999e-05 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377344+0000", 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0817e-05 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377395+0000", 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0803000000000001e-05 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377416+0000", 2024-09-13T05:20:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0993e-05 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380171+0000", 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027549660000000002 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380204+0000", 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3331999999999999e-05 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:324 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'104, mlcod=66'101)", 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.382617+0000", 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.5912013169999994, 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002752174, 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.382617+0000", 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.382617+0000", 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.382615+0000", 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383431+0000", 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081575199999999999 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383459+0000", 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7650000000000001e-05 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383471+0000", 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2578000000000001e-05 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383520+0000", 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9311000000000003e-05 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.383538+0000", 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7367000000000002e-05 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.385332+0000", 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001794192 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.385369+0000", 2024-09-13T05:20:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7064999999999998e-05 2024-09-13T05:20:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.073 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:07.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:07.075 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:07.073+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:07.176 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:07.173+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:07.247 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:07.248 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:07.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:07.306 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:07.305+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:07.366 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:20:07.377 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:07.373+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:07.383 INFO:tasks.workunit.client.0.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T05:20:07.391 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/tmp/tmp.WvgaeitFGU/mnt/test bs=1M count=1 oflag=direct 2024-09-13T05:20:07.391 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:07.391 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:07.393 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:07.393 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:20:07.395 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:20:07.395 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:07.395 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WvgaeitFGU/data of=/tmp/tmp.WvgaeitFGU/mnt/test bs=1M count=1 oflag=direct 2024-09-13T05:20:07.407 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-09-13T05:20:07.407 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-09-13T05:20:07.407 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00254467 s, 412 MB/s 2024-09-13T05:20:07.408 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo umount /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:07.408 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:07.408 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:07.409 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:07.410 INFO:tasks.workunit.client.0.smithi122.stderr:++ which umount 2024-09-13T05:20:07.411 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/umount 2024-09-13T05:20:07.411 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:07.412 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:07.438 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 23738 2024-09-13T05:20:07.438 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:20:07.438 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=23738 2024-09-13T05:20:07.438 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:20:07.438 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:07.438 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:07.439 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:07.439 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:07.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:07.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:07.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:20:07.441 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:07.441 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:07.442 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:07.442 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:07.442 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:07.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:07.479 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:07.476+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:07.530 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:20:07.531 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23738\b' 2024-09-13T05:20:07.531 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23738\b' 2024-09-13T05:20:07.579 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:07.577+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:07.582 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:07.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:20:07.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *23738\b' 2024-09-13T05:20:07.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *23738\b' 2024-09-13T05:20:07.584 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.06498673199999, 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:20:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:20:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.06471470699999, 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:20:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:20:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:20:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.064671121, 2024-09-13T05:20:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.064638894, 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:20:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:20:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:20:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:20:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:20:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:20:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.06442774600001, 2024-09-13T05:20:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:20:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.064391453, 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:20:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.06435300299999, 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.064178101, 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:20:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.249474163000002, 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.241944050000001, 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.232374723, 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:20:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.221211851, 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:20:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:20:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.215052254, 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:20:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:20:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.166216474000002, 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:20:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.157939939999999, 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:20:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:20:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:20:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:20:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:20:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.665179532, 2024-09-13T05:20:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:20:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:20:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.664744384999999, 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:20:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.664430393, 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:20:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:20:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.664025142, 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:20:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:20:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.663700603999999, 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:20:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:20:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:20:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:07.702 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:07.703 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:07.703 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:07.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:07.704 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:07.701+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:07.804 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:07.801+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:07.878 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:07.879 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:07.879 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:07.879 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:07.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:07.905 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:07.904+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:08.005 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:08.004+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:08.068 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:08.080 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:08.091 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:20:08.091 INFO:tasks.workunit.client.0.smithi122.stderr:+ cat /tmp/tmp.WvgaeitFGU/rbd-nbd.log 2024-09-13T05:20:08.092 INFO:tasks.workunit.client.0.smithi122.stdout:2024-09-13T05:20:06.026+0000 7f652a456a80 0 ceph version 18.2.4-820-g48a7de7e (48a7de7e6e1701a88fd571c7e34908ac4b8ed412) reef (stable), process rbd-nbd, pid 23729 2024-09-13T05:20:08.092 INFO:tasks.workunit.client.0.smithi122.stdout:2024-09-13T05:20:06.026+0000 7f652a456a80 0 pidfile_write: ignore empty --pid-file 2024-09-13T05:20:08.092 INFO:tasks.workunit.client.0.smithi122.stdout:2024-09-13T05:20:06.238+0000 7f6526f4d640 0 -- 172.21.15.122:0/1174838758 >> [v2:172.21.15.16:6816/1006404377,v1:172.21.15.16:6817/1006404377] conn(0x7f650802d150 msgr2=0x7f6508026010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T05:20:08.092 INFO:tasks.workunit.client.0.smithi122.stdout:2024-09-13T05:20:06.286+0000 7f6526f4d640 0 -- 172.21.15.122:0/1174838758 >> [v2:172.21.15.16:6808/2608485151,v1:172.21.15.16:6809/2608485151] conn(0x55bbc2f88540 msgr2=0x55bbc2f8a930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T05:20:08.092 INFO:tasks.workunit.client.0.smithi122.stdout:2024-09-13T05:20:07.510+0000 7f650cff9640 0 rbd-nbd: disconnect request received 2024-09-13T05:20:08.092 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep 'quiesce failed' /tmp/tmp.WvgaeitFGU/rbd-nbd.log 2024-09-13T05:20:08.092 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep 'quiesce failed' /tmp/tmp.WvgaeitFGU/rbd-nbd.log 2024-09-13T05:20:08.094 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:08.095 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd21435 2024-09-13T05:20:08.095 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:20:08.096 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:20:08.097 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:20:08.097 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:20:08.099 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:20:08.099 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:20:08.099 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd21435 2024-09-13T05:20:08.106 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:08.104+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:08.207 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:08.205+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.880847653000004, 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:08.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:08.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.869641891999997, 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:08.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:08.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.859462837999999, 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:08.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:08.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.852027034999999, 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:08.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:08.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.845419331999999, 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.234 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:08.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.836193113999997, 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:08.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4797.0:1 3.3 e48/47 3:c4c92e5a:::rbd_trash:head v 48'1, mlcod=0'0)", 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.748699199999997, 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015947019999999999, 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:08.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459810+0000", 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8649999999999998e-06 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459820+0000", 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9899999999999992e-06 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459829+0000", 2024-09-13T05:20:08.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.431e-06 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459885+0000", 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5868000000000001e-05 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459912+0000", 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7203e-05 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461364+0000", 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014515820000000001 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461399+0000", 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5063e-05 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:08.248 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.341483713999999, 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:20:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:20:08.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.208520855, 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:20:08.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:20:08.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4982.0:50 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:5 [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:08.257 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.333527042, 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001569698, 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4982", 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1259763078", 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 50 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:20:08.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874977+0000", 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874976+0000", 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967296 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874978+0000", 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2100000000000001e-06 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874981+0000", 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1760000000000002e-06 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.874984+0000", 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9569999999999998e-06 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876017+0000", 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00103304 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876260+0000", 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000243403 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.876546+0000", 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028614500000000002 2024-09-13T05:20:08.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.229960101, 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017381950000000001, 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:08.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.263 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:08.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978546+0000", 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2059999999999999e-06 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978555+0000", 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5969999999999992e-06 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978561+0000", 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7390000000000002e-06 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979976+0000", 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014147669999999999 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980108+0000", 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013226399999999999 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980282+0000", 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017327799999999999 2024-09-13T05:20:08.266 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'182, mlcod=65'180)", 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.518972027, 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016537559999999999, 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:08.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689530+0000", 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689546+0000", 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6439999999999998e-05 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689575+0000", 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9159999999999999e-05 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689590+0000", 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5056e-05 2024-09-13T05:20:08.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689724+0000", 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133518 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689793+0000", 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8769000000000005e-05 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691110+0000", 2024-09-13T05:20:08.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001317807 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691185+0000", 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4876000000000006e-05 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5233.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'183, mlcod=65'181)", 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.351162305999999, 2024-09-13T05:20:08.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001624459, 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857339+0000", 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857354+0000", 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5157000000000001e-05 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857383+0000", 2024-09-13T05:20:08.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8759e-05 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857396+0000", 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3223e-05 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857503+0000", 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010699199999999999 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857575+0000", 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1092000000000003e-05 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.858894+0000", 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001319408 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.858966+0000", 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1811999999999996e-05 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5233.0:16 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'185, mlcod=65'183)", 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.864111+0000", 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.344393, 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001563947, 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864111+0000", 2024-09-13T05:20:08.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864111+0000", 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864109+0000", 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864125+0000", 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.624e-05 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864159+0000", 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3846000000000001e-05 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864177+0000", 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7629999999999999e-05 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864276+0000", 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9622000000000003e-05 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.864338+0000", 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1805999999999997e-05 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.865604+0000", 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656670000000001 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.865675+0000", 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0739000000000002e-05 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:77 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'192, mlcod=65'190)", 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.298895336000001, 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020134300000000001, 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909606+0000", 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909624+0000", 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7799999999999999e-05 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909657+0000", 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3541999999999998e-05 2024-09-13T05:20:08.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909676+0000", 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.861e-05 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909774+0000", 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8009000000000003e-05 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909818+0000", 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4332000000000002e-05 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911530+0000", 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712129 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911622+0000", 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1228000000000001e-05 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.284 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 430 entries 65'193 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'191)", 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.294421504999999, 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018374859999999999, 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914080+0000", 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914105+0000", 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5726999999999999e-05 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914150+0000", 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4879000000000002e-05 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914188+0000", 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8099999999999998e-05 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914334+0000", 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014600099999999999 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915920+0000", 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015852329999999999 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 438 entries 65'194 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:88 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'192)", 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.27504115, 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001852292, 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933460+0000", 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933481+0000", 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0746000000000001e-05 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933524+0000", 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.278e-05 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933562+0000", 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8769000000000001e-05 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933738+0000", 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017590699999999999 2024-09-13T05:20:08.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.935315+0000", 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00157636 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:149 2.3 e66/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 66'198, mlcod=65'196)", 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.916732364000001, 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001636668, 2024-09-13T05:20:08.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291770+0000", 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291783+0000", 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3030000000000001e-05 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291808+0000", 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5075999999999998e-05 2024-09-13T05:20:08.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291820+0000", 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2092999999999999e-05 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291900+0000", 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0465999999999998e-05 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291958+0000", 2024-09-13T05:20:08.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7346000000000003e-05 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.293350+0000", 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001392784 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.293408+0000", 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7392999999999998e-05 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5242.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'201, mlcod=66'199)", 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.801162186999999, 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897480000000001, 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407339+0000", 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407360+0000", 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0703000000000002e-05 2024-09-13T05:20:08.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407422+0000", 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2426000000000002e-05 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407442+0000", 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407568+0000", 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126019 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407656+0000", 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7880999999999994e-05 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409045+0000", 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001389798 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409131+0000", 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5741999999999993e-05 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5248.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'204, mlcod=66'202)", 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.700021186000001, 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017322259999999999, 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508480+0000", 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508498+0000", 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7955999999999999e-05 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508532+0000", 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3886e-05 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508547+0000", 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.484e-05 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508657+0000", 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110522 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508720+0000", 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2345e-05 2024-09-13T05:20:08.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510147+0000", 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014275119999999999 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510215+0000", 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7402000000000003e-05 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:08.306 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:08.306 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:08.307 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:08.308 INFO:tasks.workunit.client.0.smithi122.stderr:+ OUT='/dev/nbd0 c1187b74-da7c-46e3-a569-3535787cb354' 2024-09-13T05:20:08.308 INFO:tasks.workunit.client.0.smithi122.stderr:+ read DEV COOKIE 2024-09-13T05:20:08.308 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:08.308 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:08.308 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:08.308 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:08.308 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:08.334 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23854 2024-09-13T05:20:08.334 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23854 2024-09-13T05:20:08.334 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23854 -C rbd-nbd 2024-09-13T05:20:08.341 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:08.341 INFO:tasks.workunit.client.0.smithi122.stdout: 23854 ? 00:00:00 rbd-nbd 2024-09-13T05:20:08.341 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:08.341 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:08.342 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:08.344 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:08.344 INFO:tasks.workunit.client.0.smithi122.stderr:++ which mount 2024-09-13T05:20:08.345 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/mount 2024-09-13T05:20:08.345 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:08.345 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:08.350 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:08.349+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:08.371 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device detach rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:08.371 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:08.371 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:08.372 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:08.372 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:08.373 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:08.373 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:08.374 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:08.410 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:08.408+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:08.442 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:08.442 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:08.442 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:08.443 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:08.456 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:08.470 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false get_pid rbd 2024-09-13T05:20:08.470 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:08.470 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:08.471 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:08.471 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:08.471 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:08.510 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:08.509+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:08.543 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID= 2024-09-13T05:20:08.543 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n '' 2024-09-13T05:20:08.543 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 1 2024-09-13T05:20:08.543 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:08.543 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:08.544 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:08.544 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:08.544 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:08.544 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:08.544 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:08.546 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:08.546 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:08.546 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:08.578 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: could not validate attach request 2024-09-13T05:20:08.578 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-13T05:20:08.578 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: must specify --cookie or --force to proceed 2024-09-13T05:20:08.580 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:08.580 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n c1187b74-da7c-46e3-a569-3535787cb354 ']' 2024-09-13T05:20:08.581 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 --cookie c1187b74-da7c-46e3-a569-3535787cb354 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:08.581 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:08.581 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:08.581 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:08.581 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:08.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:08.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:08.583 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --cookie c1187b74-da7c-46e3-a569-3535787cb354 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:08.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:08.609+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:08.622 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:08.622 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:08.622 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:08.622 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:08.622 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:08.636 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:08.708 INFO:tasks.workunit.client.0.smithi122.stdout:/dev/nbd0 2024-09-13T05:20:08.712 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:08.712+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:08.714 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:08.714 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:08.714 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:08.715 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:08.715 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:08.757 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:08.757 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:08.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:08.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:08.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.622507891, 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.758 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:20:08.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:20:08.760 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.761 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.621876632, 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.762 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.763 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.764 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.765 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.621231511, 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.766 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.767 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.768 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.769 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.62072414799999, 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.771 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.772 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:20:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.774 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.620198712, 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.775 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.776 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:20:08.777 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.778 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:08.779 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.61956120000001, 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:08.780 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:20:08.781 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.782 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.783 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.618984492, 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:20:08.784 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.785 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:20:08.786 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:20:08.787 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.788 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.61843301899999, 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.789 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.790 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:20:08.791 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.792 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.793 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.617972436, 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.794 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.795 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.796 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:20:08.797 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.61746255, 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:20:08.800 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.801 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.802 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.617035487, 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 118.230275769, 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:20:08.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.809 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:08.811 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 117.612740957, 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:08.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.815 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.381962022000003, 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.816 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:20:08.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.371341678, 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:20:08.820 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:20:08.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.364127363999998, 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.826 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:20:08.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:20:08.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.357771843999998, 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:20:08.832 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:20:08.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:08.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.357715480000003, 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:08.838 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:20:08.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:20:08.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:20:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.843 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.357677909000003, 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.844 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:20:08.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:20:08.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:20:08.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:20:08.848 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.848 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.848 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:08.848 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:20:08.848 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:20:08.848 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:20:08.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.357643385000003, 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:08.850 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:20:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:20:08.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:20:08.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:20:08.856 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.856 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.856 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.856 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:08.856 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:08.856 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:08.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:08.858 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23924 2024-09-13T05:20:08.858 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23924 2024-09-13T05:20:08.858 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23924 -C rbd-nbd 2024-09-13T05:20:08.859 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:08.859 INFO:tasks.workunit.client.0.smithi122.stdout: 23924 ? 00:00:00 rbd-nbd 2024-09-13T05:20:08.860 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:08.860 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:08.856+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:08.861 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-09-13T05:20:08.861 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:08.861 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:08.861 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:08.861 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:08.861 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:08.861 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:08.861 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-09-13T05:20:08.875 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false get_pid rbd 2024-09-13T05:20:08.876 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:08.876 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:08.876 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:08.876 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:08.876 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:08.945 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID= 2024-09-13T05:20:08.946 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n '' 2024-09-13T05:20:08.946 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 1 2024-09-13T05:20:08.946 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:08.946 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n c1187b74-da7c-46e3-a569-3535787cb354 ']' 2024-09-13T05:20:08.946 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 --cookie c1187b74-da7c-46e3-a569-3535787cb354 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:08.946 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:08.946 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:08.947 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:08.947 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:08.948 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:08.948 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:08.948 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --cookie c1187b74-da7c-46e3-a569-3535787cb354 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:08.958 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:08.956+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:09.011 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:09.027 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:09.031 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:09.031 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:09.031 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:09.031 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:09.043 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:09.045 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:09.059 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:09.057+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:09.061 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:09.074 INFO:tasks.workunit.client.0.smithi122.stdout:/dev/nbd0 2024-09-13T05:20:09.080 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:09.080 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:09.080 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:09.081 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:09.081 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:09.081 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:09.081 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:09.102 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:09.148 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=23982 2024-09-13T05:20:09.149 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 23982 2024-09-13T05:20:09.149 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 23982 -C rbd-nbd 2024-09-13T05:20:09.157 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:09.157 INFO:tasks.workunit.client.0.smithi122.stdout: 23982 ? 00:00:00 rbd-nbd 2024-09-13T05:20:09.157 INFO:tasks.workunit.client.0.smithi122.stderr:+ ls /tmp/tmp.WvgaeitFGU/mnt/ 2024-09-13T05:20:09.159 INFO:tasks.workunit.client.0.smithi122.stdout:lost+found 2024-09-13T05:20:09.159 INFO:tasks.workunit.client.0.smithi122.stdout:test 2024-09-13T05:20:09.159 INFO:tasks.workunit.client.0.smithi122.stderr:+ dd if=/tmp/tmp.WvgaeitFGU/mnt/test of=/dev/null bs=1M count=1 2024-09-13T05:20:09.160 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:09.160+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:09.167 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-09-13T05:20:09.167 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-09-13T05:20:09.167 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00685922 s, 153 MB/s 2024-09-13T05:20:09.167 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo dd if=/tmp/tmp.WvgaeitFGU/data of=/tmp/tmp.WvgaeitFGU/mnt/test1 bs=1M count=1 oflag=direct 2024-09-13T05:20:09.168 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:09.168 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:09.168 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:09.168 INFO:tasks.workunit.client.0.smithi122.stderr:++ which dd 2024-09-13T05:20:09.169 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/dd 2024-09-13T05:20:09.170 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:09.170 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WvgaeitFGU/data of=/tmp/tmp.WvgaeitFGU/mnt/test1 bs=1M count=1 oflag=direct 2024-09-13T05:20:09.190 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records in 2024-09-13T05:20:09.191 INFO:tasks.workunit.client.0.smithi122.stderr:1+0 records out 2024-09-13T05:20:09.191 INFO:tasks.workunit.client.0.smithi122.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154354 s, 67.9 MB/s 2024-09-13T05:20:09.191 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo umount /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:09.192 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:09.192 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:09.193 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:09.193 INFO:tasks.workunit.client.0.smithi122.stderr:++ which umount 2024-09-13T05:20:09.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/umount 2024-09-13T05:20:09.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:09.194 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:09.210 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 23982 2024-09-13T05:20:09.210 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:20:09.210 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=23982 2024-09-13T05:20:09.210 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:20:09.210 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:09.210 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:09.211 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:09.211 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:09.211 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:09.211 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:09.212 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:09.212 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:09.212 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:20:09.228 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:09.272 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:09.272+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:09.298 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:20:09.298 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^23982\b' 2024-09-13T05:20:09.298 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^23982\b' 2024-09-13T05:20:09.350 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:09.350 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:20:09.351 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *23982\b' 2024-09-13T05:20:09.351 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *23982\b' 2024-09-13T05:20:09.357 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:09.357 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:09.361 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:09.360+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:09.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:20:09.393 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:09.393 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:09.407 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:09.497 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:09.496+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:09.562 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:09.560+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:09.577 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:09.577 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:09.591 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:09.676 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:09.676+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:09.752 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:09.752 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:09.764 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:09.764+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:09.767 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:09.859 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n c1187b74-da7c-46e3-a569-3535787cb354 ']' 2024-09-13T05:20:09.859 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --show-cookie --cookie 'abc de' --options try-netlink map rbd/testrbdnbd21435 2024-09-13T05:20:09.860 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:20:09.860 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:20:09.861 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:20:09.862 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:20:09.864 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:20:09.865 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:20:09.865 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --show-cookie --cookie 'abc de' --options try-netlink map rbd/testrbdnbd21435 2024-09-13T05:20:09.865 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:09.864+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:09.932 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:09.932 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:09.946 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:09.988 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:09.988+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:10.019 INFO:tasks.workunit.client.0.smithi122.stderr:+ OUT='/dev/nbd0 abc de' 2024-09-13T05:20:10.019 INFO:tasks.workunit.client.0.smithi122.stderr:+ read DEV ANOTHER_COOKIE 2024-09-13T05:20:10.019 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:10.019 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:10.019 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:10.019 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:10.020 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:10.067 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:10.064+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:10.085 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=24056 2024-09-13T05:20:10.085 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 24056 2024-09-13T05:20:10.085 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 24056 -C rbd-nbd 2024-09-13T05:20:10.088 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:10.088 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:10.092 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:10.092 INFO:tasks.workunit.client.0.smithi122.stdout: 24056 ? 00:00:00 rbd-nbd 2024-09-13T05:20:10.093 INFO:tasks.workunit.client.0.smithi122.stderr:+ test 'abc de' = 'abc de' 2024-09-13T05:20:10.093 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd0 24056 2024-09-13T05:20:10.093 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd0 2024-09-13T05:20:10.093 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=24056 2024-09-13T05:20:10.093 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:20:10.094 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:10.094 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:10.094 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:10.094 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:10.096 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:10.096 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:10.096 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-13T05:20:10.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:10.169 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:10.168+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:10.179 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:20:10.179 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^24056\b' 2024-09-13T05:20:10.179 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^24056\b' 2024-09-13T05:20:10.221 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:10.221 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:10.226 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:10.226 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:20:10.226 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *24056\b' 2024-09-13T05:20:10.227 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *24056\b' 2024-09-13T05:20:10.233 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:10.233 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:10.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:10.282 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:10.281+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:10.353 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:10.353 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:10.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:20:10.389 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:10.385+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:10.471 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:10.469+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:10.516 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:10.516 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:10.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:10.574 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:10.573+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:10.673 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:10.672+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:10.704 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:10.704 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:10.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:10.734 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:20:10.735 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd snap ls rbd/testrbdnbd21435 2024-09-13T05:20:10.735 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-13T05:20:10.774 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:10.772+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:10.795 INFO:tasks.workunit.client.0.smithi122.stderr:+ SNAPID=4 2024-09-13T05:20:10.795 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd21435 2024-09-13T05:20:10.795 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:20:10.795 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:20:10.796 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:20:10.796 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:20:10.798 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:20:10.798 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:20:10.798 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd21435 2024-09-13T05:20:10.846 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:10.851 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:10.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:10.875 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:10.872+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:10.879 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:20:10.880 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:20:10.880 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-13T05:20:10.880 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T05:20:10.880 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T05:20:10.880 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-13T05:20:10.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ OUT='/dev/nbd0 9770834a-5002-476d-978c-713ad94db460' 2024-09-13T05:20:10.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ read DEV COOKIE 2024-09-13T05:20:10.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:10.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:10.927 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:10.927 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:10.928 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:10.976 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:10.973+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:10.991 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:10.991 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:10.993 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=24140 2024-09-13T05:20:10.993 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 24140 2024-09-13T05:20:10.993 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 24140 -C rbd-nbd 2024-09-13T05:20:11.001 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:11.001 INFO:tasks.workunit.client.0.smithi122.stdout: 24140 ? 00:00:00 rbd-nbd 2024-09-13T05:20:11.002 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device detach rbd/testrbdnbd21435 --snap-id 4 --device-type nbd 2024-09-13T05:20:11.002 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:11.002 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:11.003 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:11.003 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:11.004 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:11.004 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:11.004 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd21435 --snap-id 4 --device-type nbd 2024-09-13T05:20:11.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:11.037 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "" 2024-09-13T05:20:11.037 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:11.077 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:11.076+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:11.095 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false get_pid rbd 2024-09-13T05:20:11.095 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:11.095 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:11.095 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:11.095 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:11.096 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:11.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:11.162 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID= 2024-09-13T05:20:11.163 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n '' 2024-09-13T05:20:11.163 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 1 2024-09-13T05:20:11.163 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:11.163 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:11.163 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:11.163 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:11.163 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:11.164 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:11.165 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:11.166 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:11.166 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:11.166 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:11.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:11.198 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: could not validate attach request 2024-09-13T05:20:11.198 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-13T05:20:11.198 INFO:tasks.workunit.client.0.smithi122.stderr:rbd: must specify --cookie or --force to proceed 2024-09-13T05:20:11.200 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:11.200 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n 9770834a-5002-476d-978c-713ad94db460 ']' 2024-09-13T05:20:11.200 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device attach --device /dev/nbd0 --cookie 9770834a-5002-476d-978c-713ad94db460 --snap-id 4 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:11.201 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:11.201 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:11.201 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:11.202 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:11.203 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:11.203 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:11.203 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --cookie 9770834a-5002-476d-978c-713ad94db460 --snap-id 4 rbd/testrbdnbd21435 --device-type nbd 2024-09-13T05:20:11.214 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:11.213+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:11.272 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:11.279 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:11.277+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:11.289 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:11.289 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:11.306 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:11.306 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:11.328 INFO:tasks.workunit.client.0.smithi122.stdout:/dev/nbd0 2024-09-13T05:20:11.328 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:11.329 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:11.334 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:11.334 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:11.335 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:11.335 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:11.335 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:11.337 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.008716100000001, 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:11.338 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:11.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:11.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:11.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.997510339000002, 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:11.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:11.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:11.348 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.987331285000003, 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:11.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.352 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.979895482000003, 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:11.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.973287779000003, 2024-09-13T05:20:11.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:11.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.964061561000001, 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:11.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4797.0:1 3.3 e48/47 3:c4c92e5a:::rbd_trash:head v 48'1, mlcod=0'0)", 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.876567647000002, 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015947019999999999, 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459804+0000", 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459810+0000", 2024-09-13T05:20:11.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8649999999999998e-06 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459820+0000", 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9899999999999992e-06 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459829+0000", 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.431e-06 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459885+0000", 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5868000000000001e-05 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459912+0000", 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7203e-05 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461364+0000", 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014515820000000001 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461399+0000", 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5063e-05 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.469352161000003, 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:20:11.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:20:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.336389302000001, 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.378 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:20:11.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 25.357828548000001, 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017381950000000001, 2024-09-13T05:20:11.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978546+0000", 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2059999999999999e-06 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978555+0000", 2024-09-13T05:20:11.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5969999999999992e-06 2024-09-13T05:20:11.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978561+0000", 2024-09-13T05:20:11.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7390000000000002e-06 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979976+0000", 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014147669999999999 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980108+0000", 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013226399999999999 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980282+0000", 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017327799999999999 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'182, mlcod=65'180)", 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.646840474000001, 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016537559999999999, 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.387 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689530+0000", 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689546+0000", 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6439999999999998e-05 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689575+0000", 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9159999999999999e-05 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689590+0000", 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5056e-05 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689724+0000", 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133518 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689793+0000", 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8769000000000005e-05 2024-09-13T05:20:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691110+0000", 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001317807 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691185+0000", 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4876000000000006e-05 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.391 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5233.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'183, mlcod=65'181)", 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.479030753, 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001624459, 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857341+0000", 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857339+0000", 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857354+0000", 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5157000000000001e-05 2024-09-13T05:20:11.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857383+0000", 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8759e-05 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857396+0000", 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3223e-05 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857503+0000", 2024-09-13T05:20:11.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010699199999999999 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.857575+0000", 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1092000000000003e-05 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.858894+0000", 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001319408 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.858966+0000", 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1811999999999996e-05 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:77 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'192, mlcod=65'190)", 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.426763782999998, 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020134300000000001, 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909606+0000", 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909624+0000", 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7799999999999999e-05 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909657+0000", 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3541999999999998e-05 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909676+0000", 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.861e-05 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909774+0000", 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8009000000000003e-05 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909818+0000", 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4332000000000002e-05 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911530+0000", 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712129 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911622+0000", 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1228000000000001e-05 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 430 entries 65'193 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'191)", 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.422289952, 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018374859999999999, 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914080+0000", 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914105+0000", 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5726999999999999e-05 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914150+0000", 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4879000000000002e-05 2024-09-13T05:20:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914188+0000", 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8099999999999998e-05 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914334+0000", 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014600099999999999 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915920+0000", 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015852329999999999 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 438 entries 65'194 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:88 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'192)", 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.402909597000001, 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001852292, 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:11.404 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933460+0000", 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:11.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933481+0000", 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0746000000000001e-05 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933524+0000", 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.278e-05 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933562+0000", 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8769000000000001e-05 2024-09-13T05:20:11.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933738+0000", 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017590699999999999 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.935315+0000", 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00157636 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:149 2.3 e66/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 66'198, mlcod=65'196)", 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.044600810999999, 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001636668, 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:20:11.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291771+0000", 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291770+0000", 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291783+0000", 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3030000000000001e-05 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291808+0000", 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5075999999999998e-05 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291820+0000", 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2092999999999999e-05 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291900+0000", 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0465999999999998e-05 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.291958+0000", 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7346000000000003e-05 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.293350+0000", 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001392784 2024-09-13T05:20:11.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.293408+0000", 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7392999999999998e-05 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5242.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'201, mlcod=66'199)", 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.929030634, 2024-09-13T05:20:11.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897480000000001, 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407339+0000", 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407360+0000", 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0703000000000002e-05 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407422+0000", 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2426000000000002e-05 2024-09-13T05:20:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407442+0000", 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407568+0000", 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126019 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407656+0000", 2024-09-13T05:20:11.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7880999999999994e-05 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409045+0000", 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001389798 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409131+0000", 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5741999999999993e-05 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5248.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'204, mlcod=66'202)", 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 20.827889633000002, 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017322259999999999, 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508480+0000", 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508498+0000", 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7955999999999999e-05 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508532+0000", 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3886e-05 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508547+0000", 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.484e-05 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508657+0000", 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110522 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508720+0000", 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2345e-05 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510147+0000", 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014275119999999999 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510215+0000", 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7402000000000003e-05 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 73/16 rep_tid 573 entries 73'268 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5228.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 73'266)", 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.9667342799999998, 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686610000000001, 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369636+0000", 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369661+0000", 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4822999999999999e-05 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369714+0000", 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2991999999999997e-05 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369752+0000", 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8421999999999999e-05 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369927+0000", 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017514599999999999 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.371306+0000", 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001379171 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.424 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5228.0:83 2.3 e73/16 2:d073633c:::rbd_header.11d2ef348a4:head v 73'270, mlcod=73'268)", 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.888019817, 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712819, 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:11.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448351+0000", 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:11.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448366+0000", 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448400+0000", 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4085999999999999e-05 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448418+0000", 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7952999999999999e-05 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448539+0000", 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012074399999999999 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448602+0000", 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3207999999999995e-05 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.449986+0000", 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013839639999999999 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:11.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.450065+0000", 2024-09-13T05:20:11.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8957999999999999e-05 2024-09-13T05:20:11.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.429 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:11.429 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:11.429 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:11.430 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:11.430 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=24204 2024-09-13T05:20:11.431 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 24204 2024-09-13T05:20:11.431 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 24204 -C rbd-nbd 2024-09-13T05:20:11.431 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:11.431 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:11.431 INFO:tasks.workunit.client.0.smithi122.stdout: 24204 ? 00:00:00 rbd-nbd 2024-09-13T05:20:11.433 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:11.433 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:11.434 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-09-13T05:20:11.434 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:11.434 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:11.434 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:11.434 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:11.434 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:11.434 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:11.434 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-09-13T05:20:11.435 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:11.435 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:11.429+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:11.501 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false get_pid rbd 2024-09-13T05:20:11.501 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:11.502 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:11.502 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:11.502 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:11.502 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd0'\'']/id' 2024-09-13T05:20:11.532 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:11.532+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:11.564 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:11.564 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:11.564 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:11.564 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:11.571 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID= 2024-09-13T05:20:11.571 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n '' 2024-09-13T05:20:11.571 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 1 2024-09-13T05:20:11.571 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:11.571 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:20:11.571 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config image set rbd/testrbdnbd21435 rbd_discard_granularity_bytes 4096 2024-09-13T05:20:11.577 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:11.632 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:11.629+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:11.678 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd feature enable rbd/testrbdnbd21435 journaling 2024-09-13T05:20:11.733 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:11.732+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:11.743 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:11.743 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:11.743 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:11.743 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:11.743 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:11.758 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:11.788 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:20:11.788 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:20:11.788 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:20:11.789 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:20:11.790 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:20:11.792 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:20:11.792 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:20:11.792 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:20:11.834 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:11.833+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:11.927 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:11.927 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.79300917, 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:20:11.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.932 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.792377911, 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:20:11.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.79173279, 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:20:11.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:20:11.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:20:11.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:20:11.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:20:11.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:20:11.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:20:11.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.791225427, 2024-09-13T05:20:11.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:11.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:20:11.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.790699991, 2024-09-13T05:20:11.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:20:11.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.790062479, 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:20:11.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:20:11.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:20:11.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:20:11.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:20:11.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:20:11.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:20:11.962 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.789485771, 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:20:11.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:20:11.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.788934298, 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:11.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:20:11.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:20:11.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.78847371499999, 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:20:11.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:20:11.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:20:11.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.78796382900001, 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.976 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:20:11.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.787536766, 2024-09-13T05:20:11.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:20:11.981 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.981 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.981 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:11.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:20:11.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:11.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:20:11.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:20:11.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 121.40077704799999, 2024-09-13T05:20:11.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:11.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:20:11.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:20:11.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:20:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.991 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.992 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.992 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:20:11.992 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:11.992 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 120.78324223600001, 2024-09-13T05:20:11.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:20:11.992 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:20:11.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:11.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.552463301000003, 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:11.997 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:11.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:11.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:20:12.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.001 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.541842957, 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:20:12.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:20:12.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:20:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:20:12.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.534628642999998, 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.008 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:20:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:20:12.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.528273122999998, 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:20:12.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:20:12.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:20:12.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:20:12.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:20:12.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:20:12.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.528216759000003, 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:20:12.020 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:12.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:12.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:12.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:20:12.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:20:12.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:20:12.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:20:12.027 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.027 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.027 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.528179188000003, 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:12.028 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:12.029 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:20:12.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.029 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:12.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:12.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:20:12.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:20:12.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:20:12.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.528144664000003, 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:12.035 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:20:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.040 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.041 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.041 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:12.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:12.043 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:12.040+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:12.143 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:12.141+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:12.209 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:12.209 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:12.210 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:12.210 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:12.223 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:12.244 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:12.241+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:12.345 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:12.345 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:12.345 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:12.345 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:12.345 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:12.346 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:12.345+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:12.361 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:12.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:20:12.458 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:12.457+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:12.482 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:12.482 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:12.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:12.483 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:12.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.483 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:12.483 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:12.483 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.138900307, 2024-09-13T05:20:12.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:20:12.483 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:20:12.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:20:12.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:20:12.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:20:12.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.489 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.127019705000002, 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:20:12.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.494 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.121076006000003, 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:20:12.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.118400297999997, 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.499 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:20:12.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:20:12.503 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.113686264999998, 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:20:12.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:20:12.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.097733992000002, 2024-09-13T05:20:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:20:12.512 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.088093790000002, 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:12.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:20:12.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:20:12.517 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.081000985000003, 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:20:12.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:20:12.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.072783029, 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:12.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:12.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:20:12.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:20:12.528 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.070637216000002, 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:12.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:20:12.532 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.068719422000001, 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 44.915273159999998, 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:12.538 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:12.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:20:12.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:20:12.542 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 44.90883522, 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:12.543 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.547 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 44.902767066999999, 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:12.548 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:12.549 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:20:12.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.549 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:12.549 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.550 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.550 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:12.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.550 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:20:12.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:12.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:20:12.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:20:12.553 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 44.895223514000001, 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.554 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:20:12.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:12.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:20:12.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:20:12.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:20:12.559 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.559 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.559 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.560 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:12.560 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:12.560 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 44.882546269000002, 2024-09-13T05:20:12.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:20:12.561 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.561 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:12.561 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.561 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:12.561 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:12.562 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:20:12.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.564 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:12.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.566 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.566 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:12.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.566 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:20:12.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:12.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:20:12.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:20:12.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:20:12.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:20:12.571 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.571 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.571 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.571 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:12.571 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:12.571 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 36.395922403, 2024-09-13T05:20:12.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:20:12.572 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.572 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:12.572 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.572 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:12.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:20:12.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:12.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:20:12.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:20:12.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:20:12.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:20:12.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:20:12.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:20:12.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:20:12.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:20:12.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 36.395853582000001, 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.579 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:12.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:20:12.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:12.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:20:12.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.584 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 32.367451011999997, 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:20:12.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:20:12.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:20:12.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:20:12.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:20:12.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 23.840770698, 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:12.592 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:12.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:20:12.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:20:12.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:20:12.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:20:12.599 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.599 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.599 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.599 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:12.599 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:12.599 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:12.601 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:12.602 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:12.600+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:12.702 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:12.700+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:12.772 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:12.772 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:12.772 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:12.772 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:12.787 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:12.832 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:12.832+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:12.904 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:12.904+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:12.953 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:12.953 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:12.953 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:12.953 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:12.953 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:12.967 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:13.006 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:13.005+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:13.106 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:13.104+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:13.133 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.085705945000001, 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:13.134 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:13.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:13.136 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:13.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:13.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:13.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:13.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:13.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:13.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:13.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:13.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:13.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.140 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:13.141 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:13.141 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.049305367000002, 2024-09-13T05:20:13.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:13.141 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.141 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.141 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.141 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.141 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:13.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:13.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:13.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:13.145 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.045722062999999, 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.146 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.148 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:13.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:13.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.148 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:13.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:13.149 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.149 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.149 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.149 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:13.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:13.149 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:13.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:13.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.152 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:13.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:13.152 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.152 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.152 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.153 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.153 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.153 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:13.153 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.041087976999997, 2024-09-13T05:20:13.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.154 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:13.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.037850701000004, 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:13.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.034171301999997, 2024-09-13T05:20:13.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:13.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:13.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:13.166 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.030815109000002, 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:13.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.170 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.024483816, 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.171 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:13.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:13.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:13.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:13.175 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:13.176 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.016803867, 2024-09-13T05:20:13.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:13.176 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.176 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.176 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:13.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:13.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:13.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:13.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:13.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:13.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:13.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.014172723000001, 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:13.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:13.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:13.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:13.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:13.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:13.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:13.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.010330762999999, 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:13.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:13.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:13.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:13.190 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.191 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.191 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.191 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:13.191 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.006495366999999, 2024-09-13T05:20:13.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:13.192 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.192 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.192 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.193 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:13.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.193 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.193 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:13.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.194 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.194 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:13.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:13.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:13.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.197 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:13.198 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 38.003040120999998, 2024-09-13T05:20:13.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:13.198 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.198 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.198 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.198 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.198 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:13.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:13.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.998966281000001, 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.202 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:13.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.995887351999997, 2024-09-13T05:20:13.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:13.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:13.210 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.991644336, 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:13.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.987768334000002, 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:13.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.217 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.218 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.980142708999999, 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:13.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.221 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.222 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.976876298999997, 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.225 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 37.973183896000002, 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:13.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:13.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:13.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:13.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:13.231 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:13.231 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:13.231 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.231 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.231 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.231 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:13.231 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:13.231 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:13.233 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:13.232+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:13.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:13.333 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:13.332+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:13.369 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:13.369 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:13.369 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:13.369 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:13.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:13.390 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:20:13.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:13.458 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:13.457+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:13.506 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:13.506 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:13.506 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:13.506 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:13.506 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:13.521 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:13.534 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:13.532+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:13.607 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:13.608 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:13.624 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:13.624 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:13.635 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:13.633+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:13.640 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:13.640 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:13.640 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:13.640 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:13.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:13.661 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:13.661 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:13.681 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:13.682 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:13.702 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:13.702 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:13.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:13.761+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:13.779 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:13.779 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:13.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:13.837 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:13.836+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:13.938 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:13.937+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:13.966 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:13.966 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:13.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:14.066 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:14.065+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:14.140 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:14.140+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:14.150 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:14.150 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:14.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:14.240 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:14.240+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:14.335 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:14.336 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:14.341 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:14.337+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:14.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:14.442 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:14.441+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:14.521 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:14.522 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:14.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:14.544 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:14.544+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:14.645 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:14.641+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:14.706 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:14.706 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:14.721 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:14.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:14.761+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:14.847 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:14.845+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:14.852 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:14.852 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:14.866 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:14.948 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:14.948+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:14.987 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:14.988 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:15.002 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:15.049 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:15.045+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:15.150 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:15.149+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:15.166 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:15.166 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:15.180 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:15.251 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:15.249+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:15.298 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:15.298 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:15.313 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:15.352 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:15.349+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:15.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:20:15.430 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:15.430 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:15.446 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:15.462 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:15.461+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:15.552 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:15.549+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:15.570 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:15.570 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:15.584 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:15.654 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:15.652+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:15.707 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:15.707 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:15.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:15.722 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:15.766 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:15.765+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:15.855 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:15.853+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:15.871 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:15.871 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:15.887 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:15.891 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:15.892 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:15.903 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:15.903 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:15.906 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:15.921 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:15.921 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:15.938 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:15.938 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:15.956 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:15.956 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:15.956+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:16.037 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:16.037 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:16.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:16.037 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.985319089000001, 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:16.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:16.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.042 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.948918511000002, 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:16.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:16.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.945335206999999, 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:16.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.055 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.940701121000004, 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:16.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:16.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:16.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:16.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:16.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:16.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:16.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.937463845000003, 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.061 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:16.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.065 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.933784445999997, 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:16.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.930428253000002, 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:16.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:16.074 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.92409696, 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:16.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:16.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.916417011, 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:16.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.913785867000001, 2024-09-13T05:20:16.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:16.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.087 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.909943906999999, 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:16.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:16.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.093 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.093 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:20:16.093 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:20:16.093 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T05:20:16.093 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.236372 2024-09-13T05:20:16.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 0.2363715659740342 2024-09-13T05:20:16.095 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:16.095 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.906108510999999, 2024-09-13T05:20:16.095 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:16.095 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.095 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.095 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.098 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.098 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:16.098 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:16.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.098 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.098 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.099 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.902653264999998, 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.101 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:16.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:16.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.103 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:16.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:16.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.104 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:16.105 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.898579425000001, 2024-09-13T05:20:16.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:16.105 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.105 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.105 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.105 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.105 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:16.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:16.108 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.895500495999997, 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:16.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.89125748, 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.113 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:16.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.887381478000002, 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.117 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:16.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.121 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.879755852999999, 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:16.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.124 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:16.125 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.876489442999997, 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:16.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:16.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:16.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 40.872797040000002, 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.132 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:16.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:16.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:16.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:16.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:16.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:16.135 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.135 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.135 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.135 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:16.135 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:16.136 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:16.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:16.137 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:16.097+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:16.204 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:16.201+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:16.273 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:16.273 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:16.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:16.336 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:16.333+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:16.400 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:16.400+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:20:16.458 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:16.458 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:16.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:16.514 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:16.513+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:16.600 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:16.600+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:16.641 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:20:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.504015305, 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:20:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:20:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 118.494233775, 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:20:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:20:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:20:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:20:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.271403522, 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:20:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.264373927000001, 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:20:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:20:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.223591091999999, 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:20:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.216732618999998, 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:20:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:20:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 53.184590657999998, 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:20:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:20:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:20:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.570629183999998, 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:20:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:20:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.570467958000002, 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:80 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'83, mlcod=57'80)", 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.721730900000001, 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072315670000000004, 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915606+0000", 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915894+0000", 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028823899999999998 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915934+0000", 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9659000000000001e-05 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915953+0000", 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8660000000000001e-05 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916091+0000", 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790800000000001 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916150+0000", 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9079000000000001e-05 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922698+0000", 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065481339999999997 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922839+0000", 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140805 2024-09-13T05:20:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:81 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'84, mlcod=57'80)", 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716520406000001, 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003098127, 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920815+0000", 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920950+0000", 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502499999999999 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920989+0000", 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8779000000000003e-05 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921013+0000", 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3753000000000001e-05 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921422+0000", 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040899799999999999 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921481+0000", 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8811999999999999e-05 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923815+0000", 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002333852 2024-09-13T05:20:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923916+0000", 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101032 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:169 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'92, mlcod=65'89)", 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.306911821, 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004357662, 2024-09-13T05:20:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330424+0000", 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330542+0000", 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117475 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330570+0000", 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8036999999999999e-05 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330585+0000", 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4983e-05 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330651+0000", 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6361999999999999e-05 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330674+0000", 2024-09-13T05:20:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3064000000000001e-05 2024-09-13T05:20:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334660+0000", 2024-09-13T05:20:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039862229999999997 2024-09-13T05:20:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334784+0000", 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012329499999999999 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:170 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'93, mlcod=65'89)", 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.301715173000002, 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035268080000000002, 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335621+0000", 2024-09-13T05:20:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335688+0000", 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7006000000000002e-05 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335716+0000", 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.798e-05 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335728+0000", 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1712999999999999e-05 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335838+0000", 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110575 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335878+0000", 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9969000000000003e-05 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339094+0000", 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032155399999999998 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339149+0000", 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5685e-05 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:172 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'95, mlcod=65'89)", 2024-09-13T05:20:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.299574570000001, 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029153970000000001, 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337762+0000", 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337854+0000", 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2342999999999993e-05 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337868+0000", 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3658e-05 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337876+0000", 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5900000000000002e-06 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338010+0000", 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013445199999999999 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338063+0000", 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2762000000000001e-05 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340641+0000", 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002577663 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340679+0000", 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8172e-05 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:238 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'101, mlcod=66'99)", 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.188244552, 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048419240000000001, 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449091+0000", 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450611+0000", 2024-09-13T05:20:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015204629999999999 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450655+0000", 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3662999999999999e-05 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450676+0000", 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0573000000000001e-05 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450825+0000", 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014922399999999999 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450859+0000", 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3779999999999998e-05 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453864+0000", 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030051959999999999 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453935+0000", 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1302000000000003e-05 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:323 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'103, mlcod=66'100)", 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.260094308999999, 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029606749999999999, 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377243+0000", 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377314+0000", 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1248999999999995e-05 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377333+0000", 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924999999999999e-05 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377344+0000", 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0817e-05 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377395+0000", 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0803000000000001e-05 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377416+0000", 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0993e-05 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380171+0000", 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027549660000000002 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380204+0000", 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3331999999999999e-05 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5559.0:252 2.7 e73/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 73'147, mlcod=72'144)", 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.2400158480000005, 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037496309999999998, 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397329+0000", 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397341+0000", 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1321000000000001e-05 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397349+0000", 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2490000000000001e-06 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397396+0000", 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7172999999999997e-05 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397420+0000", 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3828999999999998e-05 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401029+0000", 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003609396 2024-09-13T05:20:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401072+0000", 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2308999999999998e-05 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:258 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 3194880~999424] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.2257868330000008, 2024-09-13T05:20:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047026610000000003, 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 258 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411550+0000", 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412658+0000", 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001108314 2024-09-13T05:20:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412672+0000", 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3682999999999999e-05 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412678+0000", 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1739999999999997e-06 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412716+0000", 2024-09-13T05:20:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7290999999999999e-05 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412768+0000", 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2710999999999997e-05 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412844+0000", 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6157999999999998e-05 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.414915+0000", 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020706159999999999 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416189+0000", 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012737110000000001 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416226+0000", 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7082000000000002e-05 2024-09-13T05:20:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416254+0000", 2024-09-13T05:20:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7840000000000001e-05 2024-09-13T05:20:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:260 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 28672~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.2046606549999996, 2024-09-13T05:20:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027961330000000001, 2024-09-13T05:20:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 260 2024-09-13T05:20:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432676+0000", 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432689+0000", 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3546999999999999e-05 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432704+0000", 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4508000000000001e-05 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432710+0000", 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9920000000000004e-06 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432774+0000", 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4343000000000004e-05 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432880+0000", 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106202 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433059+0000", 2024-09-13T05:20:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017896599999999999 2024-09-13T05:20:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434279+0000", 2024-09-13T05:20:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012192660000000001 2024-09-13T05:20:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435285+0000", 2024-09-13T05:20:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001006142 2024-09-13T05:20:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435415+0000", 2024-09-13T05:20:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130314 2024-09-13T05:20:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435473+0000", 2024-09-13T05:20:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8357999999999999e-05 2024-09-13T05:20:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:262 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 2150400~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.2045997970000002, 2024-09-13T05:20:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028404789999999999, 2024-09-13T05:20:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 262 2024-09-13T05:20:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432743+0000", 2024-09-13T05:20:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.434e-06 2024-09-13T05:20:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432749+0000", 2024-09-13T05:20:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7899999999999996e-06 2024-09-13T05:20:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432752+0000", 2024-09-13T05:20:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.552e-06 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433532+0000", 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00077913799999999997 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433639+0000", 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107366 2024-09-13T05:20:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433812+0000", 2024-09-13T05:20:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000172585 2024-09-13T05:20:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434709+0000", 2024-09-13T05:20:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089765699999999995 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435516+0000", 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080721099999999995 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435554+0000", 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7672000000000001e-05 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435579+0000", 2024-09-13T05:20:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4491999999999999e-05 2024-09-13T05:20:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:16.752 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:16.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:16.754 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:16.752+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:16.854 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:16.851+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:16.884 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:16.885 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:16.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:16.955 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:16.955+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:17.017 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:17.018 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:17.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:17.079 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:17.077+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:17.157 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.638482977, 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:20:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:20:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.63821095199999, 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:20:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:20:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.638167366, 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:20:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:20:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.638135139, 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:20:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.63792399099999, 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:20:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:20:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.637887698, 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.63784924799999, 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:20:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:20:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:20:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.637674346, 2024-09-13T05:20:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:20:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:20:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.822970408000003, 2024-09-13T05:20:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:20:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:20:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.815440295000002, 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:20:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:20:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:20:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.805870968000001, 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:20:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:20:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.794708096000001, 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:20:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.788548499000001, 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.228 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:20:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.739712719000003, 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:20:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.731436185, 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:20:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:20:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.238675777000001, 2024-09-13T05:20:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:20:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:20:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.23824063, 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:20:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:20:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.237926638000001, 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:20:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.237521387000001, 2024-09-13T05:20:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:20:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.237196849, 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:20:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.273 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:17.274 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:17.155+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:17.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:17.275 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:17.271+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:17.375 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:17.371+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.4 primary-affinity to 0.236372 (8154902) 2024-09-13T05:20:17.452 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:17.452 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:17.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:17.514 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:17.513+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:17.577 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:17.575+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:17.631 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:17.632 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:17.646 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:17.690 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:17.689+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:17.768 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.437465633000002, 2024-09-13T05:20:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.426259872000003, 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.416080817999998, 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.408645014999998, 2024-09-13T05:20:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.402037311999997, 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.392811094000002, 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.898101693999998, 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:20:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.765138835000002, 2024-09-13T05:20:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:20:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:20:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:20:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.786578081000002, 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017381950000000001, 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978546+0000", 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2059999999999999e-06 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978555+0000", 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5969999999999992e-06 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978561+0000", 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7390000000000002e-06 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979976+0000", 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014147669999999999 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980108+0000", 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013226399999999999 2024-09-13T05:20:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980282+0000", 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017327799999999999 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'182, mlcod=65'180)", 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.075590006999999, 2024-09-13T05:20:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016537559999999999, 2024-09-13T05:20:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689530+0000", 2024-09-13T05:20:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689546+0000", 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6439999999999998e-05 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689575+0000", 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9159999999999999e-05 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689590+0000", 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5056e-05 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689724+0000", 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133518 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689793+0000", 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8769000000000005e-05 2024-09-13T05:20:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691110+0000", 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001317807 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691185+0000", 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4876000000000006e-05 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:77 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'192, mlcod=65'190)", 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.855513316, 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020134300000000001, 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909606+0000", 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909624+0000", 2024-09-13T05:20:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7799999999999999e-05 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909657+0000", 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3541999999999998e-05 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909676+0000", 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.861e-05 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909774+0000", 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8009000000000003e-05 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909818+0000", 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4332000000000002e-05 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911530+0000", 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712129 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911622+0000", 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1228000000000001e-05 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 430 entries 65'193 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'191)", 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.851039485000001, 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018374859999999999, 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914080+0000", 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914105+0000", 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5726999999999999e-05 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914150+0000", 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4879000000000002e-05 2024-09-13T05:20:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914188+0000", 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8099999999999998e-05 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914334+0000", 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014600099999999999 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915920+0000", 2024-09-13T05:20:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015852329999999999 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 438 entries 65'194 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:88 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'192)", 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.831659129999998, 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001852292, 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933460+0000", 2024-09-13T05:20:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933481+0000", 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0746000000000001e-05 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933524+0000", 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.278e-05 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933562+0000", 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8769000000000001e-05 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933738+0000", 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017590699999999999 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.935315+0000", 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00157636 2024-09-13T05:20:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5242.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'201, mlcod=66'199)", 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.357780167000001, 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897480000000001, 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407339+0000", 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407360+0000", 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0703000000000002e-05 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407422+0000", 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2426000000000002e-05 2024-09-13T05:20:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407442+0000", 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407568+0000", 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126019 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407656+0000", 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7880999999999994e-05 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409045+0000", 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001389798 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409131+0000", 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5741999999999993e-05 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5248.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'204, mlcod=66'202)", 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.256639165999999, 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017322259999999999, 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508480+0000", 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508498+0000", 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7955999999999999e-05 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508532+0000", 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3886e-05 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508547+0000", 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.484e-05 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508657+0000", 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110522 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508720+0000", 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2345e-05 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510147+0000", 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014275119999999999 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510215+0000", 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7402000000000003e-05 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 73/16 rep_tid 573 entries 73'268 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5228.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 73'266)", 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.3954838130000002, 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686610000000001, 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369636+0000", 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369661+0000", 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4822999999999999e-05 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369714+0000", 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2991999999999997e-05 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369752+0000", 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8421999999999999e-05 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369927+0000", 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017514599999999999 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.371306+0000", 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001379171 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5228.0:83 2.3 e73/16 2:d073633c:::rbd_header.11d2ef348a4:head v 73'270, mlcod=73'268)", 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.3167693499999995, 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712819, 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448351+0000", 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448366+0000", 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448400+0000", 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4085999999999999e-05 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448418+0000", 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7952999999999999e-05 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448539+0000", 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012074399999999999 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448602+0000", 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3207999999999995e-05 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.449986+0000", 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013839639999999999 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.450065+0000", 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8957999999999999e-05 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5428.0:13 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'288, mlcod=74'286)", 2024-09-13T05:20:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.1326692759999997, 2024-09-13T05:20:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001716548, 2024-09-13T05:20:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632450+0000", 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632465+0000", 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.447e-05 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632498+0000", 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3759e-05 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632520+0000", 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1886999999999999e-05 2024-09-13T05:20:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632601+0000", 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1040999999999999e-05 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632634+0000", 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2279e-05 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.634102+0000", 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014686440000000001 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.634169+0000", 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6507999999999994e-05 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5428.0:19 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'291, mlcod=74'289)", 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.1186898540000003, 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001643327, 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646430+0000", 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646443+0000", 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.312e-05 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646466+0000", 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2932999999999999e-05 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646485+0000", 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8369e-05 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646569+0000", 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4132e-05 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646605+0000", 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5982000000000002e-05 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.648008+0000", 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014036020000000001 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.648075+0000", 2024-09-13T05:20:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6662000000000007e-05 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5434.0:13 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'295, mlcod=74'293)", 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.0239786049999999, 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001741678, 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741141+0000", 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741155+0000", 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3936e-05 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741185+0000", 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9373000000000002e-05 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741197+0000", 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.294e-05 2024-09-13T05:20:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741313+0000", 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115067 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741366+0000", 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3612000000000001e-05 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.742810+0000", 2024-09-13T05:20:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00144421 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.742885+0000", 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4286e-05 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:17.866 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:17.868 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:17.868 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:17.865+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:17.969 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:17.965+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:17.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:17.997 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:17.998 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:17.998 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:17.998 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:18.012 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:18.070 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:18.067+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:18.126 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:18.127 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:18.132 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:18.132 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:18.132 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:18.132 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:18.133 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:18.143 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:18.143 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:18.147 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:18.159 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:18.159 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:18.179 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:18.180 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:18.188 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:18.187+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:18.200 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:18.200 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:18.218 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:18.219 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:18.271 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:18.269+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:18.315 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:18.315 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:18.329 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:18.373 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:18.369+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:18.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:20:18.455 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:18.456 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:18.470 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:18.516 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:18.515+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:18.575 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:18.571+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:18.592 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:18.592 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:18.606 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:18.692 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:18.691+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:18.777 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:18.775+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:18.787 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:18.877 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:18.873+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:18.952 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:18.952 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:18.967 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:19.012 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:19.011+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:19.079 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:19.077+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:19.082 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:19.082 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:19.096 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:19.184 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:19.183+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:19.264 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:19.264 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:19.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:19.281 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:19.277+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:19.381 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:19.377+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:19.415 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:19.415 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:19.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:19.433 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:20:19.482 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:19.479+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:19.557 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:19.557 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:19.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:19.582 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:19.579+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:19.683 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:19.681+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:19.699 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:19.700 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:19.714 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:19.802 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:19.801+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:19.877 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:19.877 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:19.885 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:19.883+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:19.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:19.986 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:19.985+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:20.053 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:20.053 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:20.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:20.110 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:20.109+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:20.187 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:20.185+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:20.226 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:20.226 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:20.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:20.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:20.326 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:20.325+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:20.377 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:20.377 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:20.389 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:20.387+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:20.393 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:20.393 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:20.394 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:20.394 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:20.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:20.411 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:20.411 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:20.432 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:20.467 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:20.490 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:20.487+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:20.501 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:20.501 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:20.545 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:20.546 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:20.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:20.592 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:20.591+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:20.693 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:20.691+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.39649189, 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.385286129000001, 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.375107075000003, 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.367671272000003, 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.361063569000002, 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.351837351, 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.857127951000002, 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:20:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:20:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.724165092, 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:20:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:20:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:20:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.745604338, 2024-09-13T05:20:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017381950000000001, 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978546+0000", 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2059999999999999e-06 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978555+0000", 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5969999999999992e-06 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978561+0000", 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7390000000000002e-06 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979976+0000", 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014147669999999999 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980108+0000", 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013226399999999999 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980282+0000", 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017327799999999999 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'182, mlcod=65'180)", 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.034616264, 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016537559999999999, 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689530+0000", 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689546+0000", 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6439999999999998e-05 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689575+0000", 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9159999999999999e-05 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689590+0000", 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5056e-05 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689724+0000", 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133518 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689793+0000", 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8769000000000005e-05 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691110+0000", 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001317807 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691185+0000", 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4876000000000006e-05 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:77 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'192, mlcod=65'190)", 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.814539573000001, 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020134300000000001, 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909606+0000", 2024-09-13T05:20:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909624+0000", 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7799999999999999e-05 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909657+0000", 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3541999999999998e-05 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909676+0000", 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.861e-05 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909774+0000", 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8009000000000003e-05 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909818+0000", 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4332000000000002e-05 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911530+0000", 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712129 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911622+0000", 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1228000000000001e-05 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 430 entries 65'193 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'191)", 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.810065741999999, 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018374859999999999, 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914080+0000", 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914105+0000", 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5726999999999999e-05 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914150+0000", 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4879000000000002e-05 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914188+0000", 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8099999999999998e-05 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914334+0000", 2024-09-13T05:20:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014600099999999999 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915920+0000", 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015852329999999999 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 438 entries 65'194 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:88 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'192)", 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.790685387, 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001852292, 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933460+0000", 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933481+0000", 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0746000000000001e-05 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933524+0000", 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.278e-05 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933562+0000", 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8769000000000001e-05 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933738+0000", 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017590699999999999 2024-09-13T05:20:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.935315+0000", 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00157636 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5242.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'201, mlcod=66'199)", 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.316806423999999, 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897480000000001, 2024-09-13T05:20:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407339+0000", 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407360+0000", 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0703000000000002e-05 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407422+0000", 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2426000000000002e-05 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407442+0000", 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407568+0000", 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126019 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407656+0000", 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7880999999999994e-05 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409045+0000", 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001389798 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409131+0000", 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5741999999999993e-05 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5248.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'204, mlcod=66'202)", 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.215665423000001, 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017322259999999999, 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508480+0000", 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508498+0000", 2024-09-13T05:20:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7955999999999999e-05 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508532+0000", 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3886e-05 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508547+0000", 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.484e-05 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508657+0000", 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110522 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508720+0000", 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2345e-05 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510147+0000", 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014275119999999999 2024-09-13T05:20:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510215+0000", 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7402000000000003e-05 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 73/16 rep_tid 573 entries 73'268 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5228.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 73'266)", 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.35451007, 2024-09-13T05:20:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686610000000001, 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369636+0000", 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369661+0000", 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4822999999999999e-05 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369714+0000", 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2991999999999997e-05 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369752+0000", 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8421999999999999e-05 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369927+0000", 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017514599999999999 2024-09-13T05:20:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.371306+0000", 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001379171 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5228.0:83 2.3 e73/16 2:d073633c:::rbd_header.11d2ef348a4:head v 73'270, mlcod=73'268)", 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.275795606999999, 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712819, 2024-09-13T05:20:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448351+0000", 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448366+0000", 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448400+0000", 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4085999999999999e-05 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448418+0000", 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7952999999999999e-05 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448539+0000", 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012074399999999999 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448602+0000", 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3207999999999995e-05 2024-09-13T05:20:20.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.449986+0000", 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013839639999999999 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.450065+0000", 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8957999999999999e-05 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5428.0:13 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'288, mlcod=74'286)", 2024-09-13T05:20:20.806 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.0916955329999993, 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001716548, 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632450+0000", 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632465+0000", 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.447e-05 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632498+0000", 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3759e-05 2024-09-13T05:20:20.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632520+0000", 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1886999999999999e-05 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632601+0000", 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1040999999999999e-05 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632634+0000", 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2279e-05 2024-09-13T05:20:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.634102+0000", 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014686440000000001 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.634169+0000", 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6507999999999994e-05 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5428.0:19 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'291, mlcod=74'289)", 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.0777161110000009, 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001643327, 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646430+0000", 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646443+0000", 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.312e-05 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646466+0000", 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2932999999999999e-05 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646485+0000", 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8369e-05 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646569+0000", 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4132e-05 2024-09-13T05:20:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646605+0000", 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5982000000000002e-05 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.648008+0000", 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014036020000000001 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.648075+0000", 2024-09-13T05:20:20.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6662000000000007e-05 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5434.0:13 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'295, mlcod=74'293)", 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9830048619999996, 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001741678, 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:20.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741141+0000", 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:20.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741155+0000", 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3936e-05 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741185+0000", 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9373000000000002e-05 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741197+0000", 2024-09-13T05:20:20.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.294e-05 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741313+0000", 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115067 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741366+0000", 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3612000000000001e-05 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.742810+0000", 2024-09-13T05:20:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00144421 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.742885+0000", 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4286e-05 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:20.819 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:20.821 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:20.821 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:20.817+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:20.921 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:20.919+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:20.951 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:20.951 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:20.951 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:20.951 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:20.966 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:21.052 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:21.051+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:21.122 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:21.119+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:21.131 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:21.131 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:21.131 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:21.131 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:21.132 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:21.145 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:21.232 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:21.231+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:21.315 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:21.315 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:21.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:21.315 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:21.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.315 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:20:21.315 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:21.315 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.179890059, 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:21.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:20:21.317 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:20:21.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.17925880000001, 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:20:21.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.17861367899999, 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.324 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:20:21.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:20:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:21.328 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.178106316, 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.332 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.17758087999999, 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.335 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:20:21.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.176943368, 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.337 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:21.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:20:21.339 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.340 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:21.341 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.17636666000001, 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.342 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:20:21.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:20:21.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:20:21.345 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.17581518700001, 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.346 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:20:21.348 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.349 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:21.350 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.17535460400001, 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:20:21.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:20:21.353 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.354 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.355 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:20:21.355 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:21.355 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.174844718, 2024-09-13T05:20:21.355 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:20:21.355 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.355 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.355 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.355 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.355 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:21.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:20:21.357 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.358 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:20:21.359 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 131.17441765500001, 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:21.360 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.361 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.362 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 130.78765793700001, 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.364 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.365 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.365 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:21.365 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.365 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.365 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.365 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:20:21.365 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:20:21.366 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.367 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:21.368 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 130.170123125, 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:20:21.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:20:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.93934419, 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:20:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:20:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:20:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.928723845999997, 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:20:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:20:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.921509532000002, 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.383 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:21.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:20:21.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:21.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:21.388 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.915154012000002, 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:21.389 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.391 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.915097648, 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.394 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:21.395 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.396 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.397 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:20:21.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:20:21.399 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.915060077, 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:21.400 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:20:21.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.915025553, 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:20:21.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:20:21.411 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.411 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.411 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.411 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.411 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.411 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:21.413 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:21.413 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:21.411+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:21.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:20:21.514 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:21.513+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:21.581 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:21.582 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:21.582 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:21.582 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:21.595 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:21.615 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:21.613+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:21.715 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:21.711+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:21.719 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:21.719 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:21.719 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:21.719 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:21.719 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:21.733 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:21.817 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:21.813+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:21.833 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:20:21.904 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:21.904 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:21.904 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:21.904 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:21.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.904 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:21.904 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:21.904 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.561266494999998, 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:20:21.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:20:21.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.549385893, 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.910 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.913 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.914 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.543442194000001, 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.915 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:20:21.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.540766486000003, 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.919 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:20:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:20:21.923 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.536052453000003, 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:21.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.927 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.52010018, 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:21.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:20:21.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:20:21.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.510459978, 2024-09-13T05:20:21.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:20:21.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:20:21.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.503367173000001, 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:20:21.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:20:21.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.495149216999998, 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.941 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:20:21.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:20:21.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:21.945 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.493003404, 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:21.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.949 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 60.491085609999999, 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:20:21.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:20:21.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 54.337639348000003, 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:21.954 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:20:21.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:20:21.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 54.331201407999998, 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:20:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:20:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 54.325133254999997, 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:21.963 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:20:21.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:20:21.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 54.317589701999999, 2024-09-13T05:20:21.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:20:21.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.971 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 54.304912457, 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:21.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:20:21.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.818288590999998, 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:21.976 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.980 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.818219769999999, 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:21.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:20:21.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 41.789817200000002, 2024-09-13T05:20:21.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:20:21.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:20:21.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:20:21.991 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 33.263136885999998, 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:20:21.992 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:20:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:20:21.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:21.998 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:21.998 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:21.999 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:22.000 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:21.997+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:22.100 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:22.097+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:22.126 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:22.126 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:22.126 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:22.126 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:22.140 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:22.201 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:22.197+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:22.262 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:22.262 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:22.262 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:22.262 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:22.262 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:22.276 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:22.302 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:22.301+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.349999252000003, 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.398 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:22.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.400 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.402 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.313598673999998, 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:22.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:22.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.406 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.310015370000002, 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:22.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:22.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.410 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.305381283999999, 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.414 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.302144007999999, 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:22.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.418 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.298464609, 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:22.420 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.295108415999998, 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:22.424 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.425 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.426 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.288777123000003, 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:22.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.430 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.281097174000003, 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:22.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.434 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.278466029999997, 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.438 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.274624070000002, 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:22.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:22.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:22.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.270788674000002, 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:22.446 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.267333428000001, 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.447 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:22.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:22.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.263259587999997, 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.260180659, 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:22.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.457 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.458 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.255937643000003, 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:22.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.462 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.252061640999997, 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:22.466 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.244436016000002, 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:22.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:22.470 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.241169606, 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 47.237477202999997, 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:22.478 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:22.479 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:20:22.479 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T05:20:22.480 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:22.479+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:22.480 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:22.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:22.580 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:22.579+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:22.609 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:22.609 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:22.609 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:22.609 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:22.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:22.677 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:22.677 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:22.681 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:22.681+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:22.694 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:22.710 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:22.710 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:22.731 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:22.756 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:22.756 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:22.765 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:22.766 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:22.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:22.786 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:22.786 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:22.786 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:22.785+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:22.840 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T05:20:22.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T05:20:22.884 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:22.883+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:22.905 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:22.905 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:22.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:23.013 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:23.009+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:23.085 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:23.081+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:23.087 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:23.087 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:23.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:23.186 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:23.183+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:23.274 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:23.274 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:23.286 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:23.283+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:23.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:23.387 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:23.383+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:23.413 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:23.413 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:23.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:23.488 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:23.487+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:23.550 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:23.551 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:23.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:23.588 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:23.587+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:23.689 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:23.687+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:23.691 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:23.691 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:23.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:23.790 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:23.787+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:23.823 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:23.824 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:23.838 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:23.851 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T05:20:23.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:20:23.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T05:20:23.922 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:23.921+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:23.976 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:23.976 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:23.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:23.991 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:23.989+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:24.093 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:24.091+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:24.112 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:24.113 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:24.127 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:24.212 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:24.211+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:24.291 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:24.292 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:24.294 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:24.293+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:24.305 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:24.396 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:24.395+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:24.472 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:24.473 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:24.487 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:24.497 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:24.493+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:24.598 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:24.595+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:24.650 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:24.650 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:24.664 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:24.698 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:24.697+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:24.800 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:24.799+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:24.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:24.828 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:24.828 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:24.843 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T05:20:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:20:24.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T05:20:24.928 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:24.927+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:24.963 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:24.980 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:24.980 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:24.998 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:25.001 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:24.999+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:25.002 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:25.002 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:25.016 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:25.018 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:25.035 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:25.036 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:25.053 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:25.118 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:25.117+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:25.182 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:25.183 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:25.183 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:25.183 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:25.183 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:25.197 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:25.203 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:25.201+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:25.304 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:25.301+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:25.320 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:25.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.274407494999998, 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:25.321 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:25.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:25.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:25.326 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.238006917, 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:25.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:25.330 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.234423612999997, 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:25.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.334 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.335 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.229789527000001, 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:25.336 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:25.337 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.338 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.339 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.226552251000001, 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.340 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.342 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.344 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.222872852000002, 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.345 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:25.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.347 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.348 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.219516659, 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.349 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.350 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.351 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:25.352 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.213185365999998, 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.353 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:25.356 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.205505416999998, 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.357 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.358 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:25.359 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.360 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.361 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.202874272999999, 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.362 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.363 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.364 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.365 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.199032312999996, 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.366 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.367 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:25.368 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.369 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.370 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.195196916999997, 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.372 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:25.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:25.374 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.191741671000003, 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.375 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:25.376 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.377 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.378 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:25.379 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.187667830999999, 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:25.380 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.381 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.383 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.184588902000002, 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.384 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:25.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.386 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.387 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.180345885999998, 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:25.388 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:25.390 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.391 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.176469883999999, 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.393 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.394 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.395 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.396 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.168844258999997, 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:25.397 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.398 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:25.399 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.400 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.165577849000002, 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.401 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.402 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.403 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 50.161885445999999, 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:25.405 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:25.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:25.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:25.409 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:25.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:25.412 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:25.409+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:25.511 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:25.509+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:25.535 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:25.535 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:25.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:25.638 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:25.637+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:25.714 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:25.713+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:25.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:25.815 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:25.813+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:25.868 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T05:20:25.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T05:20:25.894 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 128.75981530199999, 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:20:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:20:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:20:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 127.75003377199999, 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:20:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.527203518999997, 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.520173924000005, 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.479391089000003, 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:20:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.472532615999995, 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 62.440390655000002, 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:20:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:20:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:20:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:20:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:20:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.826429181000002, 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.940 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.826267954999999, 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:20:25.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:20:25.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:20:25.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:20:25.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:25.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:80 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'83, mlcod=57'80)", 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.977530897000001, 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072315670000000004, 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:25.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915606+0000", 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915894+0000", 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028823899999999998 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915934+0000", 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9659000000000001e-05 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915953+0000", 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8660000000000001e-05 2024-09-13T05:20:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916091+0000", 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790800000000001 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916150+0000", 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9079000000000001e-05 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922698+0000", 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065481339999999997 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922839+0000", 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140805 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:81 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'84, mlcod=57'80)", 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:25.950 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.972320402999998, 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003098127, 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:25.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920815+0000", 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920950+0000", 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502499999999999 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920989+0000", 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8779000000000003e-05 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921013+0000", 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3753000000000001e-05 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921422+0000", 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040899799999999999 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921481+0000", 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8811999999999999e-05 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923815+0000", 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002333852 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923916+0000", 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101032 2024-09-13T05:20:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:169 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'92, mlcod=65'89)", 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.562711817999997, 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004357662, 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330424+0000", 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330542+0000", 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117475 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330570+0000", 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8036999999999999e-05 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330585+0000", 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4983e-05 2024-09-13T05:20:25.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330651+0000", 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6361999999999999e-05 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330674+0000", 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3064000000000001e-05 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334660+0000", 2024-09-13T05:20:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039862229999999997 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334784+0000", 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012329499999999999 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:170 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'93, mlcod=65'89)", 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.557515170000002, 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035268080000000002, 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:25.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335621+0000", 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335688+0000", 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7006000000000002e-05 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335716+0000", 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.798e-05 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335728+0000", 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1712999999999999e-05 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335838+0000", 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110575 2024-09-13T05:20:25.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335878+0000", 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9969000000000003e-05 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339094+0000", 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032155399999999998 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339149+0000", 2024-09-13T05:20:25.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5685e-05 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:172 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'95, mlcod=65'89)", 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.555374567000001, 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029153970000000001, 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337762+0000", 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337854+0000", 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2342999999999993e-05 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337868+0000", 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3658e-05 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337876+0000", 2024-09-13T05:20:25.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5900000000000002e-06 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338010+0000", 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013445199999999999 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338063+0000", 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2762000000000001e-05 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340641+0000", 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002577663 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340679+0000", 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8172e-05 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:238 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'101, mlcod=66'99)", 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.444044548999997, 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048419240000000001, 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449091+0000", 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450611+0000", 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015204629999999999 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450655+0000", 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3662999999999999e-05 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450676+0000", 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0573000000000001e-05 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450825+0000", 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014922399999999999 2024-09-13T05:20:25.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450859+0000", 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3779999999999998e-05 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453864+0000", 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030051959999999999 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453935+0000", 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1302000000000003e-05 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:323 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'103, mlcod=66'100)", 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.515894306, 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029606749999999999, 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.973 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377243+0000", 2024-09-13T05:20:25.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377314+0000", 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1248999999999995e-05 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377333+0000", 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924999999999999e-05 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377344+0000", 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0817e-05 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377395+0000", 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0803000000000001e-05 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377416+0000", 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0993e-05 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380171+0000", 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027549660000000002 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380204+0000", 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3331999999999999e-05 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5559.0:252 2.7 e73/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 73'147, mlcod=72'144)", 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.495815844999999, 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037496309999999998, 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397329+0000", 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:25.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397341+0000", 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1321000000000001e-05 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397349+0000", 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2490000000000001e-06 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397396+0000", 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7172999999999997e-05 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397420+0000", 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3828999999999998e-05 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401029+0000", 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003609396 2024-09-13T05:20:25.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401072+0000", 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2308999999999998e-05 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:258 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 3194880~999424] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.481586830000001, 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047026610000000003, 2024-09-13T05:20:25.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 258 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411550+0000", 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412658+0000", 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001108314 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412672+0000", 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3682999999999999e-05 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412678+0000", 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1739999999999997e-06 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412716+0000", 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7290999999999999e-05 2024-09-13T05:20:25.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412768+0000", 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2710999999999997e-05 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412844+0000", 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6157999999999998e-05 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.414915+0000", 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020706159999999999 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416189+0000", 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012737110000000001 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416226+0000", 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7082000000000002e-05 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416254+0000", 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7840000000000001e-05 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:260 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 28672~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.460460651999998, 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027961330000000001, 2024-09-13T05:20:25.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 260 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432676+0000", 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432689+0000", 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3546999999999999e-05 2024-09-13T05:20:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432704+0000", 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4508000000000001e-05 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432710+0000", 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9920000000000004e-06 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432774+0000", 2024-09-13T05:20:25.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4343000000000004e-05 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432880+0000", 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106202 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433059+0000", 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017896599999999999 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434279+0000", 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012192660000000001 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435285+0000", 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001006142 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435415+0000", 2024-09-13T05:20:25.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130314 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435473+0000", 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8357999999999999e-05 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:262 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 2150400~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:25.994 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.460399794000001, 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028404789999999999, 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 262 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432743+0000", 2024-09-13T05:20:25.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.434e-06 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432749+0000", 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7899999999999996e-06 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432752+0000", 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.552e-06 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433532+0000", 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00077913799999999997 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433639+0000", 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107366 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433812+0000", 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000172585 2024-09-13T05:20:25.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434709+0000", 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089765699999999995 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435516+0000", 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080721099999999995 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435554+0000", 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7672000000000001e-05 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435579+0000", 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4491999999999999e-05 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.000 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:26.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:26.003 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:25.999+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:26.103 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:26.099+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:26.181 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:26.182 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:26.182 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:26.182 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:26.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:26.203 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:26.199+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:26.304 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:26.301+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:26.315 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:26.315 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:26.315 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:26.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:26.414 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:26.413+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:26.494 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:26.494 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:26.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:26.494 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:26.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.494 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:26.494 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:26.494 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 135.97659022600001, 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:26.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:20:26.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:20:26.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.500 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 135.976318201, 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:26.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:20:26.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:20:26.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 135.97627461499999, 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:20:26.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 135.976242388, 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:20:26.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:20:26.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:20:26.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 135.97603124, 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:26.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:20:26.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:20:26.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 135.975994947, 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:26.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:20:26.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:20:26.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.531 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 135.975956497, 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:26.532 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:20:26.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:20:26.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 135.975781595, 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:26.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:20:26.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:20:26.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:20:26.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:20:26.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:20:26.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:26.544 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.161077657000007, 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:20:26.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:26.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.153547544000006, 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.551 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:20:26.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:20:26.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.143978216999997, 2024-09-13T05:20:26.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:20:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:20:26.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:20:26.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:20:26.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:20:26.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.132815344999997, 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:20:26.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:20:26.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.126655748000005, 2024-09-13T05:20:26.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:26.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.077819968, 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:20:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:20:26.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.069543433999996, 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.580 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:26.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:20:26.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.585 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.576783026000001, 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:20:26.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.576347878999997, 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:26.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:20:26.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:20:26.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.576033887000001, 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:20:26.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:20:26.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:20:26.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.575628635999998, 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:26.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:20:26.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:20:26.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.575304097999997, 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:26.610 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:26.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:20:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:26.614 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:26.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:26.616 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:26.613+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:26.716 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:26.713+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:26.789 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:26.789 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:26.789 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:26.789 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:26.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:26.846 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:26.845+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:26.882 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T05:20:26.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:20:26.905 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T05:20:26.950 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:26.949+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:26.974 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:26.974 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:26.974 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:26.974 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:26.974 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:26.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:27.019 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:27.019+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:27.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:27.122 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:27.121+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:27.155 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:27.155 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.825622017000001, 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:27.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:27.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:27.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.814416256000001, 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:27.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.166 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.804237201999996, 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:27.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:27.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.796801399000003, 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:27.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:27.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:27.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:27.177 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.790193696000003, 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:27.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:27.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:27.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:27.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:27.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:27.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:27.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.183 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.780967477999994, 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:27.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:27.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:27.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:27.188 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.286258078000003, 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:27.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:20:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.153295219, 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:27.193 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.197 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.174734465, 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017381950000000001, 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978546+0000", 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2059999999999999e-06 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978555+0000", 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5969999999999992e-06 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978561+0000", 2024-09-13T05:20:27.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7390000000000002e-06 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979976+0000", 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014147669999999999 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980108+0000", 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013226399999999999 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980282+0000", 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017327799999999999 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'182, mlcod=65'180)", 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.463746391000001, 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016537559999999999, 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.202 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689530+0000", 2024-09-13T05:20:27.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689546+0000", 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6439999999999998e-05 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689575+0000", 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9159999999999999e-05 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689590+0000", 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5056e-05 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689724+0000", 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133518 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689793+0000", 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8769000000000005e-05 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691110+0000", 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001317807 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691185+0000", 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4876000000000006e-05 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:77 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'192, mlcod=65'190)", 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.243669699999998, 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020134300000000001, 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909606+0000", 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909624+0000", 2024-09-13T05:20:27.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7799999999999999e-05 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909657+0000", 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3541999999999998e-05 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909676+0000", 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.861e-05 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909774+0000", 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8009000000000003e-05 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909818+0000", 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4332000000000002e-05 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911530+0000", 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712129 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911622+0000", 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1228000000000001e-05 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 430 entries 65'193 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'191)", 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.239195869, 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018374859999999999, 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914080+0000", 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914105+0000", 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5726999999999999e-05 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914150+0000", 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4879000000000002e-05 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914188+0000", 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8099999999999998e-05 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914334+0000", 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014600099999999999 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915920+0000", 2024-09-13T05:20:27.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015852329999999999 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 438 entries 65'194 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:88 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'192)", 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.219815513999997, 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001852292, 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933460+0000", 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933481+0000", 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0746000000000001e-05 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933524+0000", 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.278e-05 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933562+0000", 2024-09-13T05:20:27.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8769000000000001e-05 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933738+0000", 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017590699999999999 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.935315+0000", 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00157636 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5242.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'201, mlcod=66'199)", 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.745936551, 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897480000000001, 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407339+0000", 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407360+0000", 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0703000000000002e-05 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407422+0000", 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2426000000000002e-05 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407442+0000", 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407568+0000", 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126019 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407656+0000", 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7880999999999994e-05 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409045+0000", 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001389798 2024-09-13T05:20:27.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409131+0000", 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5741999999999993e-05 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5248.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'204, mlcod=66'202)", 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.644795549999998, 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017322259999999999, 2024-09-13T05:20:27.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508480+0000", 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508498+0000", 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7955999999999999e-05 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508532+0000", 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3886e-05 2024-09-13T05:20:27.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508547+0000", 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.484e-05 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508657+0000", 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110522 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508720+0000", 2024-09-13T05:20:27.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2345e-05 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510147+0000", 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014275119999999999 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510215+0000", 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7402000000000003e-05 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 73/16 rep_tid 573 entries 73'268 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5228.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 73'266)", 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.783640197, 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686610000000001, 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:27.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369636+0000", 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369661+0000", 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4822999999999999e-05 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369714+0000", 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2991999999999997e-05 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369752+0000", 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8421999999999999e-05 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369927+0000", 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017514599999999999 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.371306+0000", 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001379171 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5228.0:83 2.3 e73/16 2:d073633c:::rbd_header.11d2ef348a4:head v 73'270, mlcod=73'268)", 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.704925734, 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712819, 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448351+0000", 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448366+0000", 2024-09-13T05:20:27.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448400+0000", 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4085999999999999e-05 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448418+0000", 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7952999999999999e-05 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448539+0000", 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012074399999999999 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448602+0000", 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3207999999999995e-05 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.449986+0000", 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013839639999999999 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.450065+0000", 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8957999999999999e-05 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5428.0:13 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'288, mlcod=74'286)", 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 15.52082566, 2024-09-13T05:20:27.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001716548, 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632450+0000", 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632465+0000", 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.447e-05 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632498+0000", 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3759e-05 2024-09-13T05:20:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632520+0000", 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1886999999999999e-05 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632601+0000", 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1040999999999999e-05 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632634+0000", 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2279e-05 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.634102+0000", 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014686440000000001 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.634169+0000", 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6507999999999994e-05 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.240 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5428.0:19 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'291, mlcod=74'289)", 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 15.506846238, 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001643327, 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646430+0000", 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:27.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646443+0000", 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.312e-05 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646466+0000", 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2932999999999999e-05 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646485+0000", 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8369e-05 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646569+0000", 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4132e-05 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646605+0000", 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5982000000000002e-05 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.648008+0000", 2024-09-13T05:20:27.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014036020000000001 2024-09-13T05:20:27.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.648075+0000", 2024-09-13T05:20:27.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6662000000000007e-05 2024-09-13T05:20:27.245 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5434.0:13 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'295, mlcod=74'293)", 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 15.412134989, 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001741678, 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:27.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741141+0000", 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:27.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741155+0000", 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3936e-05 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741185+0000", 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9373000000000002e-05 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741197+0000", 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.294e-05 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741313+0000", 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115067 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741366+0000", 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3612000000000001e-05 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.742810+0000", 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00144421 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.742885+0000", 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4286e-05 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:27.250 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:27.252 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:27.253 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:27.253 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:27.254 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:27.254 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:27.249+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:27.269 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:27.286 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:27.287 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:27.306 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:27.306 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:27.353 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:27.351+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:27.382 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:27.382 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:27.396 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:27.484 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:27.483+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:27.554 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:27.551+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:27.564 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:27.564 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:27.578 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:27.655 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:27.653+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:27.744 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:27.744 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:27.756 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:27.753+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:27.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:27.858 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:27.857+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:27.880 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:27.880 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:27.891 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:20:27.893 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T05:20:27.896 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:27.920 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T05:20:27.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:20:27.962 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:27.961+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:28.017 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:28.017 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:28.032 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:28.076 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:28.075+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:28.156 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:28.156 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:28.161 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:28.159+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:28.170 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:28.262 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:28.261+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:28.265 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:20:28.284 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-13T05:20:28.083468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149562274,"num_objects":78,"num_object_clones":20,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":12,"num_read":5853,"num_read_kb":189332,"num_write":953,"num_write_kb":275306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":913,"ondisk_log_size":913,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":478556,"kb_used_data":285572,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":565752484,"statfs":{"total":579820584960,"available":579330543616,"internally_reserved":0,"allocated":292425728,"data_stored":309891497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000924"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":224,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228893+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:20:20.228893+0000","last_peered":"2024-09-13T05:20:20.228893+0000","last_clean":"2024-09-13T05:20:20.228893+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:20:20.228893+0000","last_undegraded":"2024-09-13T05:20:20.228893+0000","last_fullsized":"2024-09-13T05:20:20.228893+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"74'302","reported_seq":1955,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:22.239419+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:20:22.239419+0000","last_peered":"2024-09-13T05:20:22.239419+0000","last_clean":"2024-09-13T05:20:22.239419+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:20:22.239419+0000","last_undegraded":"2024-09-13T05:20:22.239419+0000","last_fullsized":"2024-09-13T05:20:22.239419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":16777254,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3608,"num_read_kb":23883,"num_write":312,"num_write_kb":20774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"64'5","reported_seq":66,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-13T05:20:24.328199+0000","last_change":"2024-09-13T05:19:25.480800+0000","last_active":"2024-09-13T05:20:24.328199+0000","last_peered":"2024-09-13T05:20:24.328199+0000","last_clean":"2024-09-13T05:20:24.328199+0000","last_became_active":"2024-09-13T05:19:23.455354+0000","last_became_peered":"2024-09-13T05:19:23.455354+0000","last_unstale":"2024-09-13T05:20:24.328199+0000","last_undegraded":"2024-09-13T05:20:24.328199+0000","last_fullsized":"2024-09-13T05:20:24.328199+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:20.817288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017913759999999999,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":27,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"72'64","reported_seq":345,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:21.834826+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:20:21.834826+0000","last_peered":"2024-09-13T05:20:21.834826+0000","last_clean":"2024-09-13T05:20:21.834826+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:20:21.834826+0000","last_undegraded":"2024-09-13T05:20:21.834826+0000","last_fullsized":"2024-09-13T05:20:21.834826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":25165824,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":207,"num_read_kb":27196,"num_write":60,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:21.834737+0000","last_change":"2024-09-13T05:20:18.431968+0000","last_active":"2024-09-13T05:20:21.834737+0000","last_peered":"2024-09-13T05:20:21.834737+0000","last_clean":"2024-09-13T05:20:21.834737+0000","last_became_active":"2024-09-13T05:20:18.431262+0000","last_became_peered":"2024-09-13T05:20:18.431262+0000","last_unstale":"2024-09-13T05:20:21.834737+0000","last_undegraded":"2024-09-13T05:20:21.834737+0000","last_fullsized":"2024-09-13T05:20:21.834737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:28.923680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"72'23","reported_seq":156,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-13T05:20:24.328174+0000","last_change":"2024-09-13T05:18:43.272711+0000","last_active":"2024-09-13T05:20:24.328174+0000","last_peered":"2024-09-13T05:20:24.328174+0000","last_clean":"2024-09-13T05:20:24.328174+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:20:24.328174+0000","last_undegraded":"2024-09-13T05:20:24.328174+0000","last_fullsized":"2024-09-13T05:20:24.328174+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:27.589949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388662,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":68,"num_read_kb":8204,"num_write":22,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.229883+0000","last_change":"2024-09-13T05:19:25.538085+0000","last_active":"2024-09-13T05:20:20.229883+0000","last_peered":"2024-09-13T05:20:20.229883+0000","last_clean":"2024-09-13T05:20:20.229883+0000","last_became_active":"2024-09-13T05:19:23.462578+0000","last_became_peered":"2024-09-13T05:19:23.462578+0000","last_unstale":"2024-09-13T05:20:20.229883+0000","last_undegraded":"2024-09-13T05:20:20.229883+0000","last_fullsized":"2024-09-13T05:20:20.229883+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:48.193321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056272899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"73'80","reported_seq":423,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:21.926264+0000","last_change":"2024-09-13T05:19:24.517596+0000","last_active":"2024-09-13T05:20:21.926264+0000","last_peered":"2024-09-13T05:20:21.926264+0000","last_clean":"2024-09-13T05:20:21.926264+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:20:21.926264+0000","last_undegraded":"2024-09-13T05:20:21.926264+0000","last_fullsized":"2024-09-13T05:20:21.926264+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":16781331,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":247,"num_read_kb":38365,"num_write":76,"num_write_kb":38938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"48'1","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228242+0000","last_change":"2024-09-13T05:19:25.535626+0000","last_active":"2024-09-13T05:20:20.228242+0000","last_peered":"2024-09-13T05:20:20.228242+0000","last_clean":"2024-09-13T05:20:20.228242+0000","last_became_active":"2024-09-13T05:19:23.458938+0000","last_became_peered":"2024-09-13T05:19:23.458938+0000","last_unstale":"2024-09-13T05:20:20.228242+0000","last_undegraded":"2024-09-13T05:20:20.228242+0000","last_fullsized":"2024-09-13T05:20:20.228242+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:28:08.756172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000291315,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:21.834784+0000","last_change":"2024-09-13T05:19:25.474324+0000","last_active":"2024-09-13T05:20:21.834784+0000","last_peered":"2024-09-13T05:20:21.834784+0000","last_clean":"2024-09-13T05:20:21.834784+0000","last_became_active":"2024-09-13T05:19:23.455224+0000","last_became_peered":"2024-09-13T05:19:23.455224+0000","last_unstale":"2024-09-13T05:20:21.834784+0000","last_undegraded":"2024-09-13T05:20:21.834784+0000","last_fullsized":"2024-09-13T05:20:21.834784+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:43.322756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"72'66","reported_seq":241,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:22.149460+0000","last_change":"2024-09-13T05:18:43.275214+0000","last_active":"2024-09-13T05:20:22.149460+0000","last_peered":"2024-09-13T05:20:22.149460+0000","last_clean":"2024-09-13T05:20:22.149460+0000","last_became_active":"2024-09-13T05:18:43.274886+0000","last_became_peered":"2024-09-13T05:18:43.274886+0000","last_unstale":"2024-09-13T05:20:22.149460+0000","last_undegraded":"2024-09-13T05:20:22.149460+0000","last_fullsized":"2024-09-13T05:20:22.149460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:53.129994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":16777254,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":196,"num_read_kb":16483,"num_write":80,"num_write_kb":32804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"64'1","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.229944+0000","last_change":"2024-09-13T05:20:18.435166+0000","last_active":"2024-09-13T05:20:20.229944+0000","last_peered":"2024-09-13T05:20:20.229944+0000","last_clean":"2024-09-13T05:20:20.229944+0000","last_became_active":"2024-09-13T05:20:18.434390+0000","last_became_peered":"2024-09-13T05:20:18.434390+0000","last_unstale":"2024-09-13T05:20:20.229944+0000","last_undegraded":"2024-09-13T05:20:20.229944+0000","last_fullsized":"2024-09-13T05:20:20.229944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:49:17.104799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092053199999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"73'48","reported_seq":366,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:22.239421+0000","last_change":"2024-09-13T05:19:35.195473+0000","last_active":"2024-09-13T05:20:22.239421+0000","last_peered":"2024-09-13T05:20:22.239421+0000","last_clean":"2024-09-13T05:20:22.239421+0000","last_became_active":"2024-09-13T05:19:35.077291+0000","last_became_peered":"2024-09-13T05:19:35.077291+0000","last_unstale":"2024-09-13T05:20:22.239421+0000","last_undegraded":"2024-09-13T05:20:22.239421+0000","last_fullsized":"2024-09-13T05:20:22.239421+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:50.348812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":25215090,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":217,"num_read_kb":19140,"num_write":46,"num_write_kb":41011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-13T05:20:24.328252+0000","last_change":"2024-09-13T05:20:18.431649+0000","last_active":"2024-09-13T05:20:24.328252+0000","last_peered":"2024-09-13T05:20:24.328252+0000","last_clean":"2024-09-13T05:20:24.328252+0000","last_became_active":"2024-09-13T05:20:18.430835+0000","last_became_peered":"2024-09-13T05:20:18.430835+0000","last_unstale":"2024-09-13T05:20:24.328252+0000","last_undegraded":"2024-09-13T05:20:24.328252+0000","last_fullsized":"2024-09-13T05:20:24.328252+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T05:54:07.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089178300000000005,"stat_sum":{"num_bytes":51,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"72'46","reported_seq":178,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228206+0000","last_change":"2024-09-13T05:20:18.434016+0000","last_active":"2024-09-13T05:20:20.228206+0000","last_peered":"2024-09-13T05:20:20.228206+0000","last_clean":"2024-09-13T05:20:20.228206+0000","last_became_active":"2024-09-13T05:20:18.433147+0000","last_became_peered":"2024-09-13T05:20:18.433147+0000","last_unstale":"2024-09-13T05:20:20.228206+0000","last_undegraded":"2024-09-13T05:20:20.228206+0000","last_fullsized":"2024-09-13T05:20:20.228206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:29:49.051074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777254,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":300,"num_read_kb":12518,"num_write":32,"num_write_kb":24582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228771+0000","last_change":"2024-09-13T05:19:56.340930+0000","last_active":"2024-09-13T05:20:20.228771+0000","last_peered":"2024-09-13T05:20:20.228771+0000","last_clean":"2024-09-13T05:20:20.228771+0000","last_became_active":"2024-09-13T05:19:56.340138+0000","last_became_peered":"2024-09-13T05:19:56.340138+0000","last_unstale":"2024-09-13T05:20:20.228771+0000","last_undegraded":"2024-09-13T05:20:20.228771+0000","last_fullsized":"2024-09-13T05:20:20.228771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:03:37.486649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"74'158","reported_seq":1028,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228977+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:20:20.228977+0000","last_peered":"2024-09-13T05:20:20.228977+0000","last_clean":"2024-09-13T05:20:20.228977+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:20:20.228977+0000","last_undegraded":"2024-09-13T05:20:20.228977+0000","last_fullsized":"2024-09-13T05:20:20.228977+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":23089167,"num_objects":12,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":859,"num_read_kb":43414,"num_write":160,"num_write_kb":49237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":148971836,"num_objects":70,"num_object_clones":20,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":12,"num_read":5702,"num_read_kb":189203,"num_write":788,"num_write_kb":272884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285597696,"data_stored":306545272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607584,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81772,"kb_used_data":50652,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94290068,"statfs":{"total":96636764160,"available":96553029632,"internally_reserved":0,"allocated":51867648,"data_stored":51280496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.76600000000000001,"15min":0.77600000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68300000000000005,"15min":0.68700000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0229999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.73399999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.53800000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98699999999999999,"5min":1.026,"15min":1.026},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.503,"5min":0.495,"15min":0.495},"max":{"1min":0.86199999999999999,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.52400000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.89500000000000002,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.624},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.84499999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.75700000000000001}]},{"osd":3,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":4,"up_from":54,"seq":231928233995,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70064,"kb_used_data":41952,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94301776,"statfs":{"total":96636764160,"available":96565018624,"internally_reserved":0,"allocated":42958848,"data_stored":42355790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]}]},{"osd":3,"up_from":29,"seq":124554051606,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53644,"kb_used_data":25532,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94318196,"statfs":{"total":96636764160,"available":96581832704,"internally_reserved":0,"allocated":26144768,"data_stored":25534388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64200000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.93999999999999995}]},{"osd":2,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.745}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":1.0229999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607583,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73056,"kb_used_data":37844,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94298784,"statfs":{"total":96636764160,"available":96561954816,"internally_reserved":0,"allocated":38752256,"data_stored":38172717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.621},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.61899999999999999,"15min":0.629},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.018,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.76200000000000001}]},{"osd":1,"last update":"Fri Sep 13 05:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.089,"5min":1.171,"15min":1.171},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.98099999999999998,"5min":1.21,"15min":1.21},"last":0.69899999999999995}]},{"osd":3,"last update":"Fri Sep 13 05:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.89300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":5,"last update":"Fri Sep 13 05:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.872,"15min":0.88300000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.88100000000000001,"15min":0.89700000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.161,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.0700000000000001}]}]},{"osd":1,"up_from":12,"seq":51539607583,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110600,"kb_used_data":75388,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94261240,"statfs":{"total":96636764160,"available":96523509760,"internally_reserved":0,"allocated":77197312,"data_stored":87112398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88800000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.78300000000000003}]},{"osd":2,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.58599999999999997,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.875}]},{"osd":3,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.98199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.84399999999999997,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.054,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.83199999999999996}]}]},{"osd":0,"up_from":12,"seq":51539607583,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89420,"kb_used_data":54204,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282420,"statfs":{"total":96636764160,"available":96545198080,"internally_reserved":0,"allocated":55504896,"data_stored":65435708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.71299999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.286,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.68500000000000005}]},{"osd":2,"last update":"Fri Sep 13 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.622},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80300000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.89500000000000002}]},{"osd":3,"last update":"Fri Sep 13 05:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.0009999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":5,"last update":"Fri Sep 13 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79300000000000004,"15min":0.79200000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0269999999999999,"5min":1.224,"15min":1.224},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.96199999999999997,"5min":1.694,"15min":1.694},"last":0.79300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54575104,"data_stored":65077320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75673600,"data_stored":86163642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37818368,"data_stored":37814310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42008576,"data_stored":41992344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:20:28.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:20:28.332 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:28.332 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:28.348 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:28.370 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:28.369+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:28.463 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:28.461+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:28.472 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:28.472 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:28.486 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:28.564 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:28.561+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:28.607 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:28.608 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:28.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:28.626 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:28.626 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:20:28.645 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-13T05:20:28.083468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149562274,"num_objects":78,"num_object_clones":20,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":12,"num_read":5853,"num_read_kb":189332,"num_write":953,"num_write_kb":275306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":913,"ondisk_log_size":913,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":478556,"kb_used_data":285572,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":565752484,"statfs":{"total":579820584960,"available":579330543616,"internally_reserved":0,"allocated":292425728,"data_stored":309891497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000924"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":224,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228893+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:20:20.228893+0000","last_peered":"2024-09-13T05:20:20.228893+0000","last_clean":"2024-09-13T05:20:20.228893+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:20:20.228893+0000","last_undegraded":"2024-09-13T05:20:20.228893+0000","last_fullsized":"2024-09-13T05:20:20.228893+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"74'302","reported_seq":1955,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:22.239419+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:20:22.239419+0000","last_peered":"2024-09-13T05:20:22.239419+0000","last_clean":"2024-09-13T05:20:22.239419+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:20:22.239419+0000","last_undegraded":"2024-09-13T05:20:22.239419+0000","last_fullsized":"2024-09-13T05:20:22.239419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":16777254,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3608,"num_read_kb":23883,"num_write":312,"num_write_kb":20774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"64'5","reported_seq":66,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-13T05:20:24.328199+0000","last_change":"2024-09-13T05:19:25.480800+0000","last_active":"2024-09-13T05:20:24.328199+0000","last_peered":"2024-09-13T05:20:24.328199+0000","last_clean":"2024-09-13T05:20:24.328199+0000","last_became_active":"2024-09-13T05:19:23.455354+0000","last_became_peered":"2024-09-13T05:19:23.455354+0000","last_unstale":"2024-09-13T05:20:24.328199+0000","last_undegraded":"2024-09-13T05:20:24.328199+0000","last_fullsized":"2024-09-13T05:20:24.328199+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:20.817288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017913759999999999,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":27,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"72'64","reported_seq":345,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:21.834826+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:20:21.834826+0000","last_peered":"2024-09-13T05:20:21.834826+0000","last_clean":"2024-09-13T05:20:21.834826+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:20:21.834826+0000","last_undegraded":"2024-09-13T05:20:21.834826+0000","last_fullsized":"2024-09-13T05:20:21.834826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":25165824,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":207,"num_read_kb":27196,"num_write":60,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:21.834737+0000","last_change":"2024-09-13T05:20:18.431968+0000","last_active":"2024-09-13T05:20:21.834737+0000","last_peered":"2024-09-13T05:20:21.834737+0000","last_clean":"2024-09-13T05:20:21.834737+0000","last_became_active":"2024-09-13T05:20:18.431262+0000","last_became_peered":"2024-09-13T05:20:18.431262+0000","last_unstale":"2024-09-13T05:20:21.834737+0000","last_undegraded":"2024-09-13T05:20:21.834737+0000","last_fullsized":"2024-09-13T05:20:21.834737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:28.923680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"72'23","reported_seq":156,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-13T05:20:24.328174+0000","last_change":"2024-09-13T05:18:43.272711+0000","last_active":"2024-09-13T05:20:24.328174+0000","last_peered":"2024-09-13T05:20:24.328174+0000","last_clean":"2024-09-13T05:20:24.328174+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:20:24.328174+0000","last_undegraded":"2024-09-13T05:20:24.328174+0000","last_fullsized":"2024-09-13T05:20:24.328174+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:52:27.589949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388662,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":68,"num_read_kb":8204,"num_write":22,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.229883+0000","last_change":"2024-09-13T05:19:25.538085+0000","last_active":"2024-09-13T05:20:20.229883+0000","last_peered":"2024-09-13T05:20:20.229883+0000","last_clean":"2024-09-13T05:20:20.229883+0000","last_became_active":"2024-09-13T05:19:23.462578+0000","last_became_peered":"2024-09-13T05:19:23.462578+0000","last_unstale":"2024-09-13T05:20:20.229883+0000","last_undegraded":"2024-09-13T05:20:20.229883+0000","last_fullsized":"2024-09-13T05:20:20.229883+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:48.193321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056272899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"73'80","reported_seq":423,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:21.926264+0000","last_change":"2024-09-13T05:19:24.517596+0000","last_active":"2024-09-13T05:20:21.926264+0000","last_peered":"2024-09-13T05:20:21.926264+0000","last_clean":"2024-09-13T05:20:21.926264+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:20:21.926264+0000","last_undegraded":"2024-09-13T05:20:21.926264+0000","last_fullsized":"2024-09-13T05:20:21.926264+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":16781331,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":247,"num_read_kb":38365,"num_write":76,"num_write_kb":38938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"48'1","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228242+0000","last_change":"2024-09-13T05:19:25.535626+0000","last_active":"2024-09-13T05:20:20.228242+0000","last_peered":"2024-09-13T05:20:20.228242+0000","last_clean":"2024-09-13T05:20:20.228242+0000","last_became_active":"2024-09-13T05:19:23.458938+0000","last_became_peered":"2024-09-13T05:19:23.458938+0000","last_unstale":"2024-09-13T05:20:20.228242+0000","last_undegraded":"2024-09-13T05:20:20.228242+0000","last_fullsized":"2024-09-13T05:20:20.228242+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:28:08.756172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000291315,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:21.834784+0000","last_change":"2024-09-13T05:19:25.474324+0000","last_active":"2024-09-13T05:20:21.834784+0000","last_peered":"2024-09-13T05:20:21.834784+0000","last_clean":"2024-09-13T05:20:21.834784+0000","last_became_active":"2024-09-13T05:19:23.455224+0000","last_became_peered":"2024-09-13T05:19:23.455224+0000","last_unstale":"2024-09-13T05:20:21.834784+0000","last_undegraded":"2024-09-13T05:20:21.834784+0000","last_fullsized":"2024-09-13T05:20:21.834784+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:45:43.322756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"72'66","reported_seq":241,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:22.149460+0000","last_change":"2024-09-13T05:18:43.275214+0000","last_active":"2024-09-13T05:20:22.149460+0000","last_peered":"2024-09-13T05:20:22.149460+0000","last_clean":"2024-09-13T05:20:22.149460+0000","last_became_active":"2024-09-13T05:18:43.274886+0000","last_became_peered":"2024-09-13T05:18:43.274886+0000","last_unstale":"2024-09-13T05:20:22.149460+0000","last_undegraded":"2024-09-13T05:20:22.149460+0000","last_fullsized":"2024-09-13T05:20:22.149460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:53.129994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":16777254,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":196,"num_read_kb":16483,"num_write":80,"num_write_kb":32804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"64'1","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.229944+0000","last_change":"2024-09-13T05:20:18.435166+0000","last_active":"2024-09-13T05:20:20.229944+0000","last_peered":"2024-09-13T05:20:20.229944+0000","last_clean":"2024-09-13T05:20:20.229944+0000","last_became_active":"2024-09-13T05:20:18.434390+0000","last_became_peered":"2024-09-13T05:20:18.434390+0000","last_unstale":"2024-09-13T05:20:20.229944+0000","last_undegraded":"2024-09-13T05:20:20.229944+0000","last_fullsized":"2024-09-13T05:20:20.229944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:49:17.104799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092053199999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"73'48","reported_seq":366,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T05:20:22.239421+0000","last_change":"2024-09-13T05:19:35.195473+0000","last_active":"2024-09-13T05:20:22.239421+0000","last_peered":"2024-09-13T05:20:22.239421+0000","last_clean":"2024-09-13T05:20:22.239421+0000","last_became_active":"2024-09-13T05:19:35.077291+0000","last_became_peered":"2024-09-13T05:19:35.077291+0000","last_unstale":"2024-09-13T05:20:22.239421+0000","last_undegraded":"2024-09-13T05:20:22.239421+0000","last_fullsized":"2024-09-13T05:20:22.239421+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:50.348812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":25215090,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":217,"num_read_kb":19140,"num_write":46,"num_write_kb":41011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-13T05:20:24.328252+0000","last_change":"2024-09-13T05:20:18.431649+0000","last_active":"2024-09-13T05:20:24.328252+0000","last_peered":"2024-09-13T05:20:24.328252+0000","last_clean":"2024-09-13T05:20:24.328252+0000","last_became_active":"2024-09-13T05:20:18.430835+0000","last_became_peered":"2024-09-13T05:20:18.430835+0000","last_unstale":"2024-09-13T05:20:24.328252+0000","last_undegraded":"2024-09-13T05:20:24.328252+0000","last_fullsized":"2024-09-13T05:20:24.328252+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T05:54:07.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089178300000000005,"stat_sum":{"num_bytes":51,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"72'46","reported_seq":178,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228206+0000","last_change":"2024-09-13T05:20:18.434016+0000","last_active":"2024-09-13T05:20:20.228206+0000","last_peered":"2024-09-13T05:20:20.228206+0000","last_clean":"2024-09-13T05:20:20.228206+0000","last_became_active":"2024-09-13T05:20:18.433147+0000","last_became_peered":"2024-09-13T05:20:18.433147+0000","last_unstale":"2024-09-13T05:20:20.228206+0000","last_undegraded":"2024-09-13T05:20:20.228206+0000","last_fullsized":"2024-09-13T05:20:20.228206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:29:49.051074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777254,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":300,"num_read_kb":12518,"num_write":32,"num_write_kb":24582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228771+0000","last_change":"2024-09-13T05:19:56.340930+0000","last_active":"2024-09-13T05:20:20.228771+0000","last_peered":"2024-09-13T05:20:20.228771+0000","last_clean":"2024-09-13T05:20:20.228771+0000","last_became_active":"2024-09-13T05:19:56.340138+0000","last_became_peered":"2024-09-13T05:19:56.340138+0000","last_unstale":"2024-09-13T05:20:20.228771+0000","last_undegraded":"2024-09-13T05:20:20.228771+0000","last_fullsized":"2024-09-13T05:20:20.228771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:03:37.486649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"74'158","reported_seq":1028,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T05:20:20.228977+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:20:20.228977+0000","last_peered":"2024-09-13T05:20:20.228977+0000","last_clean":"2024-09-13T05:20:20.228977+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:20:20.228977+0000","last_undegraded":"2024-09-13T05:20:20.228977+0000","last_fullsized":"2024-09-13T05:20:20.228977+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":23089167,"num_objects":12,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":859,"num_read_kb":43414,"num_write":160,"num_write_kb":49237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":148971836,"num_objects":70,"num_object_clones":20,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":12,"num_read":5702,"num_read_kb":189203,"num_write":788,"num_write_kb":272884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41943059,"num_keys_recovered":11,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285597696,"data_stored":306545272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607584,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81772,"kb_used_data":50652,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94290068,"statfs":{"total":96636764160,"available":96553029632,"internally_reserved":0,"allocated":51867648,"data_stored":51280496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.76600000000000001,"15min":0.77600000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68300000000000005,"15min":0.68700000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0229999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.73399999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.53800000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98699999999999999,"5min":1.026,"15min":1.026},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.503,"5min":0.495,"15min":0.495},"max":{"1min":0.86199999999999999,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.52400000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.89500000000000002,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.624},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.84499999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.75700000000000001}]},{"osd":3,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":4,"up_from":54,"seq":231928233995,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70064,"kb_used_data":41952,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94301776,"statfs":{"total":96636764160,"available":96565018624,"internally_reserved":0,"allocated":42958848,"data_stored":42355790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]}]},{"osd":3,"up_from":29,"seq":124554051606,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53644,"kb_used_data":25532,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94318196,"statfs":{"total":96636764160,"available":96581832704,"internally_reserved":0,"allocated":26144768,"data_stored":25534388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64200000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.93999999999999995}]},{"osd":2,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.745}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":1.0229999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607583,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73056,"kb_used_data":37844,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94298784,"statfs":{"total":96636764160,"available":96561954816,"internally_reserved":0,"allocated":38752256,"data_stored":38172717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.621},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.61899999999999999,"15min":0.629},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.018,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.76200000000000001}]},{"osd":1,"last update":"Fri Sep 13 05:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.089,"5min":1.171,"15min":1.171},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.98099999999999998,"5min":1.21,"15min":1.21},"last":0.69899999999999995}]},{"osd":3,"last update":"Fri Sep 13 05:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.89300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":5,"last update":"Fri Sep 13 05:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.872,"15min":0.88300000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.88100000000000001,"15min":0.89700000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.161,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.0700000000000001}]}]},{"osd":1,"up_from":12,"seq":51539607583,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110600,"kb_used_data":75388,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94261240,"statfs":{"total":96636764160,"available":96523509760,"internally_reserved":0,"allocated":77197312,"data_stored":87112398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88800000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.78300000000000003}]},{"osd":2,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.58599999999999997,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.875}]},{"osd":3,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.98199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.84399999999999997,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.054,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.83199999999999996}]}]},{"osd":0,"up_from":12,"seq":51539607583,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89420,"kb_used_data":54204,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282420,"statfs":{"total":96636764160,"available":96545198080,"internally_reserved":0,"allocated":55504896,"data_stored":65435708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.71299999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.286,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.68500000000000005}]},{"osd":2,"last update":"Fri Sep 13 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.622},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80300000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.89500000000000002}]},{"osd":3,"last update":"Fri Sep 13 05:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.0009999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":5,"last update":"Fri Sep 13 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79300000000000004,"15min":0.79200000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0269999999999999,"5min":1.224,"15min":1.224},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.96199999999999997,"5min":1.694,"15min":1.694},"last":0.79300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54575104,"data_stored":65077320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75673600,"data_stored":86163642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37818368,"data_stored":37814310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":25165916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42008576,"data_stored":41992344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:20:28.647 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T05:20:28.647 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:20:28.647 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5] 2024-09-13T05:20:28.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-09-13T05:20:28.665 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:28.663+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:28.741 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:28.741 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:28.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:28.765 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:28.763+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:28.866 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:28.865+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:28.873 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:28.873 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:28.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:28.967 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:28.963+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:28.986 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-09-13T05:20:28.981+0000 7fb6535af640 -1 mon.a@0(leader).osd e86 definitely_dead 0 2024-09-13T05:20:29.056 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:29.056 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:29.068 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:29.065+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:29.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:29.169 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:29.167+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:29.236 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:29.236 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:29.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:29.294 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:29.293+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:29.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:29.371 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:29.369+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:29.372 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:29.372 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:29.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:29.474 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:29.475 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:29.475 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:29.473+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:29.490 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:29.506 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:29.507 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:29.526 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:29.526 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:29.546 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:29.547 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:29.556 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:29.556 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:29.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:29.556 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 139.03828827500001, 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:20:29.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:20:29.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:20:29.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 139.03801625, 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:29.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:20:29.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:20:29.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:20:29.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 139.03797266399999, 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.570 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:20:29.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:20:29.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 139.037940437, 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:20:29.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:20:29.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 139.037729289, 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.582 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:20:29.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:20:29.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 139.037692996, 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:29.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:20:29.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 139.037654546, 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.594 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:20:29.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 139.037479644, 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:20:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:29.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:29.605 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.222775705999993, 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:29.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:20:29.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:20:29.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:29.611 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.215245593000006, 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:20:29.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.205676265999998, 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:20:29.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:20:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:20:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:29.621 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.194513393999998, 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:20:29.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.188353797000005, 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:29.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:20:29.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.139518017, 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:29.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:20:29.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:20:29.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.131241482999997, 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.643 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.638481075000001, 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:20:29.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:20:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.638045927999997, 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:20:29.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:29.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.655 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.637731936000002, 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:29.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:20:29.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.637326684999998, 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:29.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:20:29.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.637002146999997, 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:29.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:20:29.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:29.672 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:29.672 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:29.673 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:29.674 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:29.675 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:29.671+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:29.777 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:29.771+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:29.845 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:29.846 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:29.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:marked out osd.3. 2024-09-13T05:20:29.903 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:29.901+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:29.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:20:29.978 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:29.977+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:29.988 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:29.989 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:30.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:30.094 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:30.093+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.833667016000007, 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:30.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:30.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:30.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:30.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.822461254999993, 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.812282201000002, 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:30.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:30.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.804846397999995, 2024-09-13T05:20:30.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:30.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:30.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:30.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:30.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:30.184 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.798238694999995, 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:30.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:30.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:30.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:30.190 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.789012477, 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:30.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:30.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:30.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4679.0:502 2.0 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 62.294303077000002, 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016506120000000001, 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4679", 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:30.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 502 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867020+0000", 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867019+0000", 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867024+0000", 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5220000000000001e-06 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867038+0000", 2024-09-13T05:20:30.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4472e-05 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867046+0000", 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2039999999999997e-06 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867861+0000", 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081485100000000001 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.867971+0000", 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109658 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:27.868670+0000", 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069955699999999996 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5262.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:30.199 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.161340217999999, 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001715919, 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5262", 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2752581529", 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999983+0000", 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999982+0000", 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999984+0000", 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1440000000000001e-06 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:43.999996+0000", 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1637000000000001e-05 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.000001+0000", 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8300000000000003e-06 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001406+0000", 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00140481 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001524+0000", 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011811099999999999 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:44.001699+0000", 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017479099999999999 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.203 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4997.0:49 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [read 135168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.182779463999999, 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017381950000000001, 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4997", 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/810432878", 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 49 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.204 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978543+0000", 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978546+0000", 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2059999999999999e-06 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978555+0000", 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5969999999999992e-06 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.978561+0000", 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7390000000000002e-06 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.979976+0000", 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014147669999999999 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980108+0000", 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013226399999999999 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:45.980282+0000", 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017327799999999999 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:13 2.3 e65/16 2:d073633c:::rbd_header.11d2ef348a4:head v 65'182, mlcod=65'180)", 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.47179139, 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016537559999999999, 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689532+0000", 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689530+0000", 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689546+0000", 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6439999999999998e-05 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689575+0000", 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9159999999999999e-05 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689590+0000", 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5056e-05 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689724+0000", 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000133518 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.689793+0000", 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8769000000000005e-05 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691110+0000", 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001317807 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.691185+0000", 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4876000000000006e-05 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.212 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:77 2.3 e65/16 2:cb05b277:::rbd_object_map.11d2ef348a4:head v 65'192, mlcod=65'190)", 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.251714698999997, 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020134300000000001, 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909608+0000", 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909606+0000", 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909624+0000", 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7799999999999999e-05 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909657+0000", 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3541999999999998e-05 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909676+0000", 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.861e-05 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909774+0000", 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8009000000000003e-05 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.909818+0000", 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4332000000000002e-05 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911530+0000", 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712129 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.911622+0000", 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1228000000000001e-05 2024-09-13T05:20:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 430 entries 65'193 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'191)", 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.247240867999999, 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018374859999999999, 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914082+0000", 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914080+0000", 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914105+0000", 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5726999999999999e-05 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914150+0000", 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4879000000000002e-05 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914188+0000", 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8099999999999998e-05 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.914334+0000", 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014600099999999999 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915920+0000", 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015852329999999999 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.220 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 65/16 rep_tid 438 entries 65'194 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5227.0:88 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 65'192)", 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.227860513000003, 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001852292, 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933462+0000", 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933460+0000", 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933481+0000", 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0746000000000001e-05 2024-09-13T05:20:30.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933524+0000", 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.278e-05 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933562+0000", 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8769000000000001e-05 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.933738+0000", 2024-09-13T05:20:30.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017590699999999999 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.935315+0000", 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00157636 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5242.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'201, mlcod=66'199)", 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.753981549999999, 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017897480000000001, 2024-09-13T05:20:30.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407341+0000", 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407339+0000", 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407360+0000", 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0703000000000002e-05 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407422+0000", 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2426000000000002e-05 2024-09-13T05:20:30.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407442+0000", 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407568+0000", 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126019 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.407656+0000", 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7880999999999994e-05 2024-09-13T05:20:30.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409045+0000", 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001389798 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.409131+0000", 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5741999999999993e-05 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.228 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5248.0:13 2.3 e66/16 2:d073633c:::rbd_header.11d2ef348a4:head v 66'204, mlcod=66'202)", 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.652840548999997, 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017322259999999999, 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508482+0000", 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508480+0000", 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508498+0000", 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7955999999999999e-05 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508532+0000", 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3886e-05 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508547+0000", 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.484e-05 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508657+0000", 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110522 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.508720+0000", 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2345e-05 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510147+0000", 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014275119999999999 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.510215+0000", 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7402000000000003e-05 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "pg_update_log_missing(2.3 epoch 73/16 rep_tid 573 entries 73'268 (0'0) error 2:d073633c:::rbd_header.11d2ef348a4:head by client.5228.0:79 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 73'266)", 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.791685196, 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686610000000001, 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369638+0000", 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369636+0000", 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369661+0000", 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4822999999999999e-05 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369714+0000", 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2991999999999997e-05 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369752+0000", 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8421999999999999e-05 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.369927+0000", 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017514599999999999 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.371306+0000", 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001379171 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5228.0:83 2.3 e73/16 2:d073633c:::rbd_header.11d2ef348a4:head v 73'270, mlcod=73'268)", 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.712970732999999, 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001712819, 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448352+0000", 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448351+0000", 2024-09-13T05:20:30.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448366+0000", 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448400+0000", 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4085999999999999e-05 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448418+0000", 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7952999999999999e-05 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448539+0000", 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012074399999999999 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.448602+0000", 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3207999999999995e-05 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.449986+0000", 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013839639999999999 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:08.450065+0000", 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8957999999999999e-05 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5428.0:13 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'288, mlcod=74'286)", 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:30.240 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.528870658999999, 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001716548, 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632452+0000", 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632450+0000", 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632465+0000", 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.447e-05 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632498+0000", 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3759e-05 2024-09-13T05:20:30.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632520+0000", 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1886999999999999e-05 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632601+0000", 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1040999999999999e-05 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.632634+0000", 2024-09-13T05:20:30.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2279e-05 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.634102+0000", 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014686440000000001 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.634169+0000", 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6507999999999994e-05 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5428.0:19 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'291, mlcod=74'289)", 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.514891237000001, 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001643327, 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646432+0000", 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646430+0000", 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646443+0000", 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.312e-05 2024-09-13T05:20:30.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646466+0000", 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2932999999999999e-05 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646485+0000", 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8369e-05 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646569+0000", 2024-09-13T05:20:30.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4132e-05 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.646605+0000", 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5982000000000002e-05 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.648008+0000", 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014036020000000001 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.648075+0000", 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6662000000000007e-05 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5434.0:13 2.3 e74/16 2:d073633c:::rbd_header.11d2ef348a4:head v 74'295, mlcod=74'293)", 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.420179988000001, 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001741678, 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741143+0000", 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741141+0000", 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741155+0000", 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3936e-05 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741185+0000", 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9373000000000002e-05 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741197+0000", 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.294e-05 2024-09-13T05:20:30.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741313+0000", 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115067 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.741366+0000", 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3612000000000001e-05 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.742810+0000", 2024-09-13T05:20:30.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00144421 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:11.742885+0000", 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4286e-05 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:30.253 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:30.255 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:30.255 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:30.253+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:30.355 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:30.353+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:30.385 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:30.385 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:30.385 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:30.385 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:30.399 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:30.457 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:30.453+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:30.521 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:30.521 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:30.521 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:30.521 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:30.521 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:30.535 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:30.584 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:30.583+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:30.659 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:30.657+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:30.662 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:30.662 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:30.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:30.662 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.52722709, 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:20:30.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.52659583100001, 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.667 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:20:30.671 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.52595070999999, 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:30.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.52544334699999, 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:20:30.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:20:30.678 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.52491791099999, 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.680 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:20:30.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:20:30.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:30.684 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.52428039899999, 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.688 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.52370369100001, 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:30.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:20:30.692 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.52315221800001, 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:20:30.693 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:20:30.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:20:30.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.522691635, 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:20:30.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.700 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.701 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.522181749, 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.702 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.705 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:20:30.706 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.52175468600001, 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:30.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.710 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 140.134994968, 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.711 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 139.517460156, 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.715 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.716 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.717 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.718 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.719 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.286681220999995, 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.720 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.721 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.722 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:20:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.276060877000006, 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:30.724 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.725 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:20:30.726 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:20:30.727 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:20:30.728 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.729 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.268846562999997, 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.730 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.731 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.732 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:20:30.733 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:20:30.734 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.262491042999997, 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:20:30.735 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.736 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.738 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:20:30.739 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:20:30.740 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.262434678999995, 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:30.741 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.742 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.743 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.744 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.745 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.746 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.262397108000002, 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.747 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.748 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:20:30.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:20:30.750 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.751 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:30.752 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.262362584000002, 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.753 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:20:30.754 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:30.755 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:30.756 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:30.757 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:20:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:20:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:30.758 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:30.758 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:30.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:30.760 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:30.757+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:30.860 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:30.857+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:30.912 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:20:30.939 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:30.940 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:30.940 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:30.940 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:30.957 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:30.986 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:30.985+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:31.062 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:31.061+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:31.099 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:31.099 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:31.099 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:31.099 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:31.099 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:31.114 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:31.164 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:31.163+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:31.241 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.898116560000005, 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.242 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:31.243 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.244 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.245 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:20:31.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.247 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.886235958, 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.248 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.249 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.880292259000001, 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:20:31.252 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:20:31.256 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.877616551000003, 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.257 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.258 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.259 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.872902518000004, 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:20:31.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.263 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.856950244999993, 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.265 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:20:31.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.267 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.268 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:20:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.847310042999993, 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:31.270 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.271 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.272 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.273 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.840217237999994, 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.274 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:20:31.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:20:31.276 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.277 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.831999281999998, 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.278 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:31.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:20:31.280 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.281 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.829853469, 2024-09-13T05:20:31.282 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.283 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:31.284 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:20:31.285 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.286 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 69.827935675000006, 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.287 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.288 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:20:31.289 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:20:31.290 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 63.674489413000003, 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:31.291 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.292 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:20:31.293 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.294 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 63.668051472999998, 2024-09-13T05:20:31.295 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.296 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:20:31.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:20:31.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.299 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 63.661983319999997, 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.300 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.301 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.302 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.303 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 63.654439767, 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.304 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.305 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.306 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:20:31.307 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 63.641762522000001, 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:31.308 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.309 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.310 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:20:31.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 55.155138655999998, 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:20:31.312 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.313 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.314 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.315 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.316 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 55.155069834999999, 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:20:31.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.318 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.319 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.320 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.321 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 51.126667265000002, 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.322 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.323 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:20:31.324 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.325 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.326 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 42.599986950999998, 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:20:31.328 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.329 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.331 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.332 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.333 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.334 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.334 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:31.335 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:31.336 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:31.333+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:31.436 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:31.435+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:31.502 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:31.502 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:31.502 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:31.516 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:31.560 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:31.559+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:31.638 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:31.635+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:31.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:31.682 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:31.682 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:31.682 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:31.683 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:31.683 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:31.696 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:31.740 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:31.739+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:31.832 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:31.832 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:31.839 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:31.837+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:31.848 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:31.865 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:31.865 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:31.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:31.865 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:31.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.865 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:31.865 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:31.865 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.817105486999999, 2024-09-13T05:20:31.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:31.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:31.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.780704909000001, 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:31.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.777121604999998, 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.874 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:31.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.772487519000002, 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.878 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.880 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.881 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.769250243000002, 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:31.882 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.883 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:31.885 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.765570844000003, 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:31.886 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.887 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:31.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:31.889 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.762214651000001, 2024-09-13T05:20:31.890 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.891 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:31.892 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.893 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.894 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.755883357999998, 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:31.895 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.896 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.897 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:31.898 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.748203408999998, 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:31.899 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.900 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.901 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.902 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.745572265, 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:31.903 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.904 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.905 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.906 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.741730304999997, 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.908 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.909 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.910 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.737894908999998, 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:31.911 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.913 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.914 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.734439663000003, 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.915 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:31.916 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.917 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:31.918 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.730365823, 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:31.919 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.920 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.921 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:31.922 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.727286894000002, 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:31.923 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.924 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.925 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:31.926 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.723043877999999, 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:31.927 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:31.928 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.929 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:31.930 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.719167876, 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:31.932 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.933 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:31.934 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.711542250999997, 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.935 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:31.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:31.937 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.708275841000003, 2024-09-13T05:20:31.938 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:31.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:31.941 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 56.704583438, 2024-09-13T05:20:31.942 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.943 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.944 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:31.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:31.946 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:31.947 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:31.948 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:31.948 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:31.948 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:31.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:31.949 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:31.950 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:31.951 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:31.951 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:31.945+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:32.048 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:32.045+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:32.121 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:32.121 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:32.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:32.149 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:32.147+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:32.250 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:32.249+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:32.305 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:32.305 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:32.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:32.362 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:32.361+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:32.451 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:32.451+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:32.483 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:32.483 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:32.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:32.552 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:32.549+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:32.615 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:32.615 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:32.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:32.670 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:32.669+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:32.753 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:32.749+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:32.795 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:32.795 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:32.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:32.854 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:32.851+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:32.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:20:32.983 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:32.983 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:32.984 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:32.981+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:32.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:33.055 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:33.051+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:33.156 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:33.155+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:33.165 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:33.165 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:33.180 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:33.257 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:33.253+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:33.310 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:33.310 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:33.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:33.358 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:33.355+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:33.459 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:33.455+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:33.493 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:33.493 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:33.507 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:33.560 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:33.557+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:33.653 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:33.661 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:33.657+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:33.762 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:33.761+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:33.769 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:33.769 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:33.783 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:33.863 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:33.859+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:33.907 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:33.907 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:33.922 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:33.929 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:20:33.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:33.964 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:33.963+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:34.065 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:34.063+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:34.087 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:34.087 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:34.101 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:34.104 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:34.104 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:34.120 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:34.136 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:34.137 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:34.156 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:34.156 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:34.165 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:34.165+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:34.174 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:34.193 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:34.194 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:34.266 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:34.265+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:34.267 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:34.267 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:34.267 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:34.267 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:34.267 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:34.281 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:34.367 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:34.365+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:34.404 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.062815916999995, 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:34.405 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.406 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:20:34.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.408 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.409 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.410 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.050935315000004, 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:34.411 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.412 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.413 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:20:34.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.044991616000004, 2024-09-13T05:20:34.415 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:20:34.417 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.418 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:20:34.419 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.042315908000006, 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.420 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.421 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:20:34.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:20:34.423 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:34.424 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.037601874999993, 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:20:34.426 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.427 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.428 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.021649601999997, 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.429 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.430 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:20:34.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.432 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:20:34.433 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.012009399999997, 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:34.434 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.435 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:20:34.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:20:34.437 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.004916594999997, 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.438 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:20:34.440 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.441 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.442 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.996698639000002, 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.443 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:20:34.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:20:34.445 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.446 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.994552826000003, 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.447 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:20:34.448 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.449 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:20:34.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.992635031999995, 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:20:34.453 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.454 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.839188770000007, 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:20:34.456 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.457 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.457 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.457 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:34.457 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:34.458 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:20:34.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.458 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.458 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.458 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:34.458 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.458 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:20:34.459 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.460 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.461 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.832750829999995, 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:34.462 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.463 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:20:34.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:20:34.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.466 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.826682676999994, 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.467 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:34.468 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:20:34.469 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.470 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.819139124000003, 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.472 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:20:34.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:20:34.474 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 66.806461878999997, 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.476 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:34.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.478 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.479 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 58.319838013000002, 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.480 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:20:34.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 58.319769192000003, 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:34.485 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:20:34.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.489 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 54.291366621999998, 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:20:34.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:20:34.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:20:34.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:20:34.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:34.496 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 45.764686308000002, 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:34.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:20:34.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:20:34.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:20:34.502 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.503 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.503 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.503 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:34.504 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:34.505 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:34.503+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:34.605 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:34.605+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:34.674 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:34.674 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:34.674 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:34.674 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:34.688 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:34.706 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:34.705+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:34.807 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:34.805+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:34.811 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:34.811 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:34.811 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:34.812 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:34.812 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:34.826 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:34.907 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:34.903+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:34.931 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:20:34.931 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:20:34.931 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-13T05:20:34.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-09-13T05:20:34.946 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:34.946 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:34.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:34.946 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:34.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.946 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:34.946 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:34.946 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.898312527999998, 2024-09-13T05:20:34.946 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.947 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:34.948 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.949 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:34.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.86191195, 2024-09-13T05:20:34.951 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:34.953 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.954 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.955 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.858328645999997, 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.956 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.959 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.853694560000001, 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.960 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:34.961 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.962 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.963 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.850457284000001, 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.964 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.965 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:34.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:34.967 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.846777885000002, 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.968 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:34.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.970 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.971 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.843421692, 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.972 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.973 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.974 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.975 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.837090398999997, 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:34.976 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.977 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:34.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.979 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:34.980 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.829410449999997, 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:34.981 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.982 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.983 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.984 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.826779305999999, 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.985 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.986 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:34.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.988 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.822937346000003, 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.989 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:34.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.991 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.992 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.819101949999997, 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.994 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.995 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:34.996 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.815646704000002, 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:34.997 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.998 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:34.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:35.000 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.811572863999999, 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:35.001 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:35.002 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:35.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.002 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:35.002 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:35.002 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.002 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:35.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:35.004 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:35.005 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.808493935000001, 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:35.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:35.007 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:35.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.804250918999998, 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:35.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:35.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:35.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:35.014 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.014 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:35.014 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.014 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.800374916999999, 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:35.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:35.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.792749292000003, 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:35.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.022 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.789482882000001, 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:35.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:35.026 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 59.785790478999999, 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:35.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:35.030 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.031 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:35.031 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.031 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:35.031 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:35.031 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:35.032 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:35.031+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:35.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:35.133 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:35.131+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:35.169 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:35.169 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:35.169 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:35.170 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:35.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:35.234 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:35.233+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:35.248 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-09-13T05:20:35.245+0000 7fb6535af640 -1 mon.a@0(leader).osd e89 definitely_dead 0 2024-09-13T05:20:35.307 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:35.308 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:35.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:35.336 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:35.331+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:35.437 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:35.433+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.35637776499999, 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:20:35.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:20:35.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:20:35.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:35.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:20:35.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:20:35.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:35.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:20:35.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:20:35.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 137.34659623499999, 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:20:35.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:20:35.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:35.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.511 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.123765981999995, 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:20:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:20:35.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.116736387000003, 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:20:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:20:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:35.521 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.075953552000001, 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:20:35.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:35.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.069095078999993, 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:20:35.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:20:35.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.036953118, 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:20:35.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:20:35.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:20:35.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:20:35.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.422991644, 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:35.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.544 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.422830417999997, 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:20:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:80 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'83, mlcod=57'80)", 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.574093359999999, 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072315670000000004, 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915606+0000", 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915894+0000", 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028823899999999998 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915934+0000", 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9659000000000001e-05 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915953+0000", 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8660000000000001e-05 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916091+0000", 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790800000000001 2024-09-13T05:20:35.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916150+0000", 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9079000000000001e-05 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922698+0000", 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065481339999999997 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922839+0000", 2024-09-13T05:20:35.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140805 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:81 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'84, mlcod=57'80)", 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.568882866000003, 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003098127, 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920815+0000", 2024-09-13T05:20:35.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920950+0000", 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502499999999999 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920989+0000", 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8779000000000003e-05 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921013+0000", 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3753000000000001e-05 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921422+0000", 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040899799999999999 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921481+0000", 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8811999999999999e-05 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923815+0000", 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002333852 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923916+0000", 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101032 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.558 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:169 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'92, mlcod=65'89)", 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.159274281000002, 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004357662, 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:35.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330424+0000", 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330542+0000", 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117475 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330570+0000", 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8036999999999999e-05 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330585+0000", 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4983e-05 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330651+0000", 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6361999999999999e-05 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330674+0000", 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3064000000000001e-05 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334660+0000", 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039862229999999997 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334784+0000", 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012329499999999999 2024-09-13T05:20:35.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:170 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'93, mlcod=65'89)", 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.154077633, 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035268080000000002, 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335621+0000", 2024-09-13T05:20:35.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335688+0000", 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7006000000000002e-05 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335716+0000", 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.798e-05 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335728+0000", 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1712999999999999e-05 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335838+0000", 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110575 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335878+0000", 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9969000000000003e-05 2024-09-13T05:20:35.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339094+0000", 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032155399999999998 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339149+0000", 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5685e-05 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:172 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'95, mlcod=65'89)", 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.151937029999999, 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029153970000000001, 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337762+0000", 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337854+0000", 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2342999999999993e-05 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337868+0000", 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3658e-05 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337876+0000", 2024-09-13T05:20:35.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5900000000000002e-06 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338010+0000", 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013445199999999999 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338063+0000", 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2762000000000001e-05 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340641+0000", 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002577663 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340679+0000", 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8172e-05 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:238 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'101, mlcod=66'99)", 2024-09-13T05:20:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.040607012000002, 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048419240000000001, 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449091+0000", 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450611+0000", 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015204629999999999 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450655+0000", 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3662999999999999e-05 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450676+0000", 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0573000000000001e-05 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450825+0000", 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014922399999999999 2024-09-13T05:20:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450859+0000", 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3779999999999998e-05 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453864+0000", 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030051959999999999 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453935+0000", 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1302000000000003e-05 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:323 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'103, mlcod=66'100)", 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.112456768999998, 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029606749999999999, 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.577 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377243+0000", 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377314+0000", 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1248999999999995e-05 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377333+0000", 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924999999999999e-05 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377344+0000", 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0817e-05 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377395+0000", 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0803000000000001e-05 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377416+0000", 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0993e-05 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380171+0000", 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027549660000000002 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380204+0000", 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3331999999999999e-05 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.581 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5559.0:252 2.7 e73/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 73'147, mlcod=72'144)", 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.092378308000001, 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037496309999999998, 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397329+0000", 2024-09-13T05:20:35.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397341+0000", 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1321000000000001e-05 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397349+0000", 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2490000000000001e-06 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397396+0000", 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7172999999999997e-05 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397420+0000", 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3828999999999998e-05 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401029+0000", 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003609396 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401072+0000", 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2308999999999998e-05 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:258 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 3194880~999424] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.078149292999999, 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047026610000000003, 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 258 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411550+0000", 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412658+0000", 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001108314 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412672+0000", 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3682999999999999e-05 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412678+0000", 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1739999999999997e-06 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412716+0000", 2024-09-13T05:20:35.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7290999999999999e-05 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412768+0000", 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2710999999999997e-05 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412844+0000", 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6157999999999998e-05 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:35.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.414915+0000", 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020706159999999999 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416189+0000", 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012737110000000001 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416226+0000", 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7082000000000002e-05 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416254+0000", 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7840000000000001e-05 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:260 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 28672~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.057023115, 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027961330000000001, 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.592 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 260 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432676+0000", 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432689+0000", 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3546999999999999e-05 2024-09-13T05:20:35.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432704+0000", 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4508000000000001e-05 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432710+0000", 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9920000000000004e-06 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432774+0000", 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4343000000000004e-05 2024-09-13T05:20:35.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432880+0000", 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106202 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433059+0000", 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017896599999999999 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434279+0000", 2024-09-13T05:20:35.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012192660000000001 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435285+0000", 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001006142 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435415+0000", 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130314 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435473+0000", 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8357999999999999e-05 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:262 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 2150400~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.056962256999999, 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028404789999999999, 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:35.598 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 262 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432743+0000", 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.434e-06 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432749+0000", 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7899999999999996e-06 2024-09-13T05:20:35.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432752+0000", 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.552e-06 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433532+0000", 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00077913799999999997 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:35.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433639+0000", 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107366 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433812+0000", 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000172585 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434709+0000", 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089765699999999995 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435516+0000", 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080721099999999995 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435554+0000", 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7672000000000001e-05 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435579+0000", 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4491999999999999e-05 2024-09-13T05:20:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.604 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:35.604 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:35.604 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:35.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:35.606 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:35.605+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:35.706 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:35.703+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:35.779 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:35.779 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:35.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:35.807 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:35.807+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:35.908 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:35.905+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:35.919 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:35.919 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:35.919 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:35.919 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:35.919 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:35.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:marked in osd.3. 2024-09-13T05:20:35.957 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-13T05:20:35.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:20:36.046 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:36.045+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:36.111 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:36.107+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:36.123 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:36.123 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:36.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:36.123 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:36.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.123 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 145.60441238799999, 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:20:36.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:20:36.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 145.604140363, 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.130 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.135 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 145.604096777, 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:36.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:20:36.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:20:36.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 145.60406455, 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.142 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:36.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:20:36.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:20:36.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 145.603853402, 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:36.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 145.603817109, 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:20:36.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:20:36.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 145.603778659, 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:20:36.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:20:36.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:20:36.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:36.166 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 145.603603757, 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:36.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:20:36.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:36.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.172 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.788899818999994, 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:36.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:20:36.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:20:36.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.781369706000007, 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:20:36.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.771800378999998, 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:20:36.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:20:36.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:36.189 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.760637506999998, 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:36.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:20:36.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.195 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.754477910000006, 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:20:36.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:20:36.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.705642130000001, 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:20:36.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:20:36.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.697365595999997, 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:20:36.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:20:36.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:36.212 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.204605188000002, 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:20:36.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:20:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:20:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:20:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:20:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:20:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.204170040999998, 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:20:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.203856049000002, 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:20:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.203450797999999, 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:20:36.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:20:36.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.203126259999998, 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:36.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:20:36.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:36.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:36.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:20:36.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:20:36.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.241 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:36.241 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:36.241 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:36.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:36.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:36.244 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:36.241+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:36.343 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:36.341+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:36.420 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:36.421 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:36.421 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:36.421 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:36.424 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:36.424 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:36.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:36.441 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:36.441 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:36.458 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:36.458 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:36.457+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:36.478 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:36.499 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:36.499 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:36.519 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:36.545 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:36.543+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:36.557 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:36.557 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:36.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:36.645 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:36.643+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:36.739 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:36.739 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:36.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:36.745+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:36.753 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:36.847 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:36.843+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:36.884 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:36.885 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:36.899 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:36.951 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:36.945+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:36.951 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:20:37.049 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:37.047+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:37.082 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:37.082 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:37.096 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:37.150 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:37.149+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:37.251 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:37.247+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:37.263 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:37.263 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:37.278 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:37.352 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:37.349+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:37.439 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:37.440 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:37.452 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:37.449+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:37.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:37.553 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:37.551+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:37.571 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:37.571 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:37.585 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:37.654 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:37.653+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:37.754 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:37.754 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:37.755 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:37.753+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:37.768 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:37.857 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:37.855+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:37.887 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:37.887 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:37.901 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:37.988 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:37.987+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:38.059 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:38.057+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:38.064 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:38.064 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:38.078 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:38.164 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:38.163+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:38.240 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:38.240 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:38.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:38.296 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:38.295+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:38.361 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:38.359+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:38.390 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:38.390 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:38.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:38.462 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:38.459+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:38.530 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:38.530 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:38.538 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:38.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:38.566 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:38.563+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:38.667 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:38.665+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:38.680 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:38.680 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:38.697 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:38.697 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:38.713 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:38.714 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 141.57935459399999, 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:20:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:20:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 140.569573064, 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:20:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:20:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:20:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.346742810999999, 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:20:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.339713216000007, 2024-09-13T05:20:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:20:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:20:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.298930381000005, 2024-09-13T05:20:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:20:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.292071907999997, 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 75.259929947000003, 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:20:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:20:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 62.645968473000003, 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:20:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 62.645807247, 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:20:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:20:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:80 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'83, mlcod=57'80)", 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 48.797070189000003, 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072315670000000004, 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915606+0000", 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915894+0000", 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028823899999999998 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915934+0000", 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9659000000000001e-05 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915953+0000", 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8660000000000001e-05 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916091+0000", 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790800000000001 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916150+0000", 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9079000000000001e-05 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922698+0000", 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065481339999999997 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922839+0000", 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140805 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:81 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'84, mlcod=57'80)", 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 48.791859694999999, 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003098127, 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920815+0000", 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920950+0000", 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502499999999999 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920989+0000", 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8779000000000003e-05 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921013+0000", 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3753000000000001e-05 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921422+0000", 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040899799999999999 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921481+0000", 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8811999999999999e-05 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923815+0000", 2024-09-13T05:20:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002333852 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923916+0000", 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101032 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:169 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'92, mlcod=65'89)", 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 48.382251109999999, 2024-09-13T05:20:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004357662, 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330424+0000", 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330542+0000", 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117475 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330570+0000", 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8036999999999999e-05 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330585+0000", 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4983e-05 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330651+0000", 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6361999999999999e-05 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330674+0000", 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3064000000000001e-05 2024-09-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334660+0000", 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039862229999999997 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334784+0000", 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012329499999999999 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:170 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'93, mlcod=65'89)", 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 48.377054461999997, 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035268080000000002, 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335621+0000", 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335688+0000", 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7006000000000002e-05 2024-09-13T05:20:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335716+0000", 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.798e-05 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335728+0000", 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1712999999999999e-05 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335838+0000", 2024-09-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110575 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335878+0000", 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9969000000000003e-05 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339094+0000", 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032155399999999998 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339149+0000", 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5685e-05 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:172 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'95, mlcod=65'89)", 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 48.374913859000003, 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029153970000000001, 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337762+0000", 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337854+0000", 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2342999999999993e-05 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337868+0000", 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3658e-05 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337876+0000", 2024-09-13T05:20:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5900000000000002e-06 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338010+0000", 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013445199999999999 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338063+0000", 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2762000000000001e-05 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340641+0000", 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002577663 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340679+0000", 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8172e-05 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:238 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'101, mlcod=66'99)", 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 48.263583840999999, 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048419240000000001, 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449091+0000", 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450611+0000", 2024-09-13T05:20:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015204629999999999 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450655+0000", 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3662999999999999e-05 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450676+0000", 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0573000000000001e-05 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450825+0000", 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014922399999999999 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450859+0000", 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3779999999999998e-05 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453864+0000", 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030051959999999999 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453935+0000", 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1302000000000003e-05 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:323 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'103, mlcod=66'100)", 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.335433598000002, 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029606749999999999, 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377243+0000", 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377314+0000", 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1248999999999995e-05 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377333+0000", 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924999999999999e-05 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377344+0000", 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0817e-05 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377395+0000", 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0803000000000001e-05 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377416+0000", 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0993e-05 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380171+0000", 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027549660000000002 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380204+0000", 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3331999999999999e-05 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5559.0:252 2.7 e73/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 73'147, mlcod=72'144)", 2024-09-13T05:20:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.315355137000001, 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037496309999999998, 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397329+0000", 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397341+0000", 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1321000000000001e-05 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397349+0000", 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2490000000000001e-06 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397396+0000", 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7172999999999997e-05 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397420+0000", 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3828999999999998e-05 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401029+0000", 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003609396 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401072+0000", 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2308999999999998e-05 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:258 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 3194880~999424] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.301126121999999, 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047026610000000003, 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 258 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411550+0000", 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412658+0000", 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001108314 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412672+0000", 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3682999999999999e-05 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412678+0000", 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1739999999999997e-06 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412716+0000", 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7290999999999999e-05 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412768+0000", 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2710999999999997e-05 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412844+0000", 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6157999999999998e-05 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.414915+0000", 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020706159999999999 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416189+0000", 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012737110000000001 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416226+0000", 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7082000000000002e-05 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416254+0000", 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7840000000000001e-05 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:260 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 28672~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.279999944, 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027961330000000001, 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 260 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432676+0000", 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432689+0000", 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3546999999999999e-05 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432704+0000", 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4508000000000001e-05 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432710+0000", 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9920000000000004e-06 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432774+0000", 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4343000000000004e-05 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432880+0000", 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106202 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433059+0000", 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017896599999999999 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434279+0000", 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012192660000000001 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435285+0000", 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001006142 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435415+0000", 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130314 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435473+0000", 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8357999999999999e-05 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:262 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 2150400~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.279939085999999, 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028404789999999999, 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 262 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432743+0000", 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.434e-06 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432749+0000", 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7899999999999996e-06 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432752+0000", 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.552e-06 2024-09-13T05:20:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433532+0000", 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00077913799999999997 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433639+0000", 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107366 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433812+0000", 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000172585 2024-09-13T05:20:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434709+0000", 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089765699999999995 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435516+0000", 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080721099999999995 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435554+0000", 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7672000000000001e-05 2024-09-13T05:20:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435579+0000", 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4491999999999999e-05 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:38.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:38.822 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:38.823 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:38.824 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:38.819+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:38.922 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:38.921+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:38.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:20:38.996 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:38.996 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:38.996 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:38.996 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:39.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:39.024 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:39.019+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:39.125 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:39.121+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:39.190 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:39.191 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:39.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:39.246 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:39.245+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:39.327 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:39.325+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.85402787699999, 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:20:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:20:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.85375585200001, 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.853712266, 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:20:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.85368003900001, 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.85346889100001, 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.85343259800001, 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.85339414800001, 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.85321924600001, 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:20:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:20:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.038515308000001, 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.030985195, 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:20:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:20:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.021415868000005, 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:20:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:20:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.010252996000006, 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:20:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.004093398999998, 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:20:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.955257618999994, 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:20:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:20:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.946981085000004, 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:20:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:20:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:20:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:20:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.454220677000002, 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:20:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:20:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:20:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.453785529999998, 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:20:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:20:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.453471538000002, 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:20:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.453066286999999, 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:20:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 49.452741748999998, 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.487 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:39.489 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:39.489 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:39.487+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:39.590 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:39.587+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:39.661 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:39.661 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:39.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:39.691 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:39.687+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:39.792 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:39.787+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:39.798 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:39.798 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:39.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:39.893 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:39.891+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:39.939 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.610238353, 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.599032592, 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.588853537999995, 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.581417735000002, 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.574810032000002, 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.565583814000007, 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9505103179999992, 2024-09-13T05:20:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003044452, 2024-09-13T05:20:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987382+0000", 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988524+0000", 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011418859999999999 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988545+0000", 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0483e-05 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988556+0000", 2024-09-13T05:20:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1586999999999999e-05 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988662+0000", 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106137 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988680+0000", 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7295999999999998e-05 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.990428+0000", 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017486789999999999 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9405789769999995, 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002020534, 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998323+0000", 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010079329999999999 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998341+0000", 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8077e-05 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998351+0000", 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0570999999999999e-05 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998429+0000", 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7350999999999999e-05 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998439+0000", 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0651e-05 2024-09-13T05:20:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.999336+0000", 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000896469 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9361386090000003, 2024-09-13T05:20:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021888099999999998, 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001755+0000", 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002748+0000", 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099270400000000007 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002765+0000", 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7218e-05 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002776+0000", 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0783e-05 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002832+0000", 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6181999999999998e-05 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002842+0000", 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5950000000000005e-06 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.003944+0000", 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011029569999999999 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9308342649999997, 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00206362, 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007059+0000", 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008052+0000", 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099299999999999996 2024-09-13T05:20:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008070+0000", 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7742999999999999e-05 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008081+0000", 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0920999999999999e-05 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008137+0000", 2024-09-13T05:20:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5773000000000003e-05 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008146+0000", 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6469999999999993e-06 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.009124+0000", 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097727500000000011 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9218250220000002, 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024079549999999998, 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016068+0000", 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017401+0000", 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013326989999999999 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017440+0000", 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9623000000000003e-05 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017469+0000", 2024-09-13T05:20:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8776e-05 2024-09-13T05:20:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017538+0000", 2024-09-13T05:20:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9375999999999994e-05 2024-09-13T05:20:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017548+0000", 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8050000000000001e-06 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.018477+0000", 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00092904299999999995 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9167454750000008, 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030823669999999999, 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021147+0000", 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022523+0000", 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013751779999999999 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022563+0000", 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0009999999999998e-05 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022591+0000", 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8683e-05 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022701+0000", 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110173 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022721+0000", 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9911e-05 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.024231+0000", 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001509886 2024-09-13T05:20:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9076907209999998, 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031121460000000001, 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030202+0000", 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031555+0000", 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001352518 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031595+0000", 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0197000000000002e-05 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031646+0000", 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0742000000000003e-05 2024-09-13T05:20:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031787+0000", 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014145999999999999 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031806+0000", 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8845e-05 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.033316+0000", 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015096739999999999 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9014792319999998, 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025441959999999999, 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036414+0000", 2024-09-13T05:20:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037751+0000", 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013372289999999999 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037788+0000", 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7023000000000001e-05 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037814+0000", 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6308999999999998e-05 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037903+0000", 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8930000000000004e-05 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037915+0000", 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1474000000000001e-05 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.038959+0000", 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010445610000000001 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8959829310000007, 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025004459999999999, 2024-09-13T05:20:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041910+0000", 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043262+0000", 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001352645 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043304+0000", 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1746000000000003e-05 2024-09-13T05:20:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043335+0000", 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0618999999999999e-05 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043438+0000", 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000103261 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043451+0000", 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.261e-05 2024-09-13T05:20:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.044412+0000", 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00096126200000000005 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8901726199999995, 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038240729999999999, 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047720+0000", 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049066+0000", 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013460589999999999 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049110+0000", 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3702999999999998e-05 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049137+0000", 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7443000000000001e-05 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049277+0000", 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013948399999999999 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049298+0000", 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1234e-05 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.051546+0000", 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022478400000000001 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8796301450000001, 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025842819999999998, 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058262+0000", 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059597+0000", 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001334256 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059635+0000", 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8562000000000001e-05 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059662+0000", 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6703000000000002e-05 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059751+0000", 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9163e-05 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059762+0000", 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0671e-05 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.060848+0000", 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001086697 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8738613389999994, 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030803480000000001, 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064032+0000", 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065378+0000", 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013465630000000001 2024-09-13T05:20:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065418+0000", 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0422e-05 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065446+0000", 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7477000000000001e-05 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065577+0000", 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131254 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065599+0000", 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1656999999999999e-05 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.067113+0000", 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001514409 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8675992210000008, 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030880719999999999, 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070294+0000", 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071663+0000", 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001369089 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071704+0000", 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0745999999999999e-05 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071731+0000", 2024-09-13T05:20:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7182000000000002e-05 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071865+0000", 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013386800000000001 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071885+0000", 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0800000000000001e-05 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.073383+0000", 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001497757 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8612510069999999, 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033570940000000001, 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076642+0000", 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077619+0000", 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097751200000000004 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077660+0000", 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0368999999999999e-05 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077690+0000", 2024-09-13T05:20:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0139999999999999e-05 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077823+0000", 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013348400000000001 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077842+0000", 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8255999999999999e-05 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.080000+0000", 2024-09-13T05:20:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021586169999999998 2024-09-13T05:20:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:40.030 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:40.032 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:20:40.033 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:40.027+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:40.133 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:40.131+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:40.161 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:40.161 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:40.162 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:40.162 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:40.175 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:40.234 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:40.233+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:40.335 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:40.333+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:40.338 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:40.338 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:40.338 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:40.338 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:40.338 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:40.353 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:40.435 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:40.431+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:40.480 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:40.480 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:40.480 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:40.480 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.344971829, 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.481 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:20:40.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:20:40.483 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.484 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.34434057000001, 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:40.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:20:40.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:40.489 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.34369544899999, 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.493 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.343188086, 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.494 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:20:40.495 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:20:40.496 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.498 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.34266264999999, 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.501 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:20:40.503 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.342025138, 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:20:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:20:40.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.34144843000001, 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.508 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:20:40.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.34089695700001, 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:20:40.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.516 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.34043637400001, 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:20:40.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:20:40.520 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.339926488, 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:40.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:20:40.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:20:40.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:20:40.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:20:40.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 150.33949942500001, 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:20:40.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 149.95273970700001, 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:20:40.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:20:40.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:20:40.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:20:40.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:20:40.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.534 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 149.335204895, 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:20:40.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.10442596, 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.093805615999997, 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.543 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:40.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:20:40.548 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.086591302000002, 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:20:40.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.550 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:20:40.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:40.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.554 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.080235782000003, 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.555 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:20:40.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:40.560 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.080179418, 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:20:40.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:20:40.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:20:40.565 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.080141846999993, 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:40.566 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:20:40.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:20:40.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.571 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.080107322999993, 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:40.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:40.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:40.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:40.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:40.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:20:40.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:20:40.577 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.577 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.577 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.577 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:40.577 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:40.577 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:40.579 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:40.579 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:40.575+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:40.679 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:40.677+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:40.747 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:40.747 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:40.748 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:40.748 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:40.761 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:40.808 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:40.807+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:40.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:40.881 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:40.879+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:40.883 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:40.883 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:40.884 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:40.884 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:40.884 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:40.898 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:40.957 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:20:40.958 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:20:40.958 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T05:20:40.958 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T05:20:40.983 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:40.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:40.988 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:40.987+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:41.000 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:41.017 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:41.017 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:41.034 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:41.055 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:41.075 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:41.076 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:41.076 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:41.082 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:41.081+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:41.091 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:41.183 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:41.181+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:41.250 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:41.250 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:41.250 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 92, 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2024-09-13T05:17:57.363809+0000", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2024-09-13T05:20:39.951098+0000", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2024-09-13T05:19:34.047702+0000", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2024-09-13T05:20:35.926417+0000", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5806080, 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "nodeep-scrub", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "noscrub", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2024-09-13T05:20:41.251 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 7, 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 3, 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "reef", 2024-09-13T05:20:41.252 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_crimson": false, 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": ".mgr", 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:08.122164+0000", 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:20:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:20:41.254 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:20:41.254 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:20:41.254 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:20:41.254 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:20:41.254 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2024-09-13T05:20:41.254 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2024-09-13T05:20:41.254 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2024-09-13T05:20:41.254 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2024-09-13T05:20:41.254 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2024-09-13T05:20:41.255 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:20:41.255 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:20:41.255 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:20:41.255 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:20:41.255 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:20:41.255 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:20:41.255 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:20:41.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.255 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "15", 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:20:41.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:20:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:20:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": {} 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.259 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 6.059999942779541, 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 6.059999942779541, 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.33000001311302185, 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2, 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2, 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.86952722072601318, 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2024-09-13T05:20:41.260 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:18:15.001664+0000", 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:20:41.261 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "73", 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 8, 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 73, 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:20:41.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:20:41.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:20:41.265 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 2.25, 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 2.25, 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 1, 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2.25, 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2.25, 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.86952722072601318, 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-13T05:20:41.266 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 3, 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "new_default_pool21435", 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-13T05:19:21.793970+0000", 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-13T05:20:41.267 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-13T05:20:41.268 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "50", 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 3, 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 50, 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-13T05:20:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-13T05:20:41.270 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-13T05:20:41.271 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 1.5, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 1.5, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 1, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 1.5, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 1.5, 2024-09-13T05:20:41.272 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.86952722072601318, 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "e58174f4-85ec-4321-8bc3-384261c59c6b", 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:41.273 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 77, 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.274 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2024-09-13T05:20:41.277 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2153409514 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/2153409514", 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/2153409514", 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/2153409514", 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/2153409514", 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:41.278 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "84a3db6c-dacd-43d5-b964-37fdef0091a4", 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:20:41.279 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 90, 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2024-09-13T05:20:41.280 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:41.283 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2608485151 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/2608485151", 2024-09-13T05:20:41.284 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/2608485151", 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/2608485151", 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/2608485151", 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "0d530476-cf38-4544-9ce7-e8ed35df220b", 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:41.285 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 90, 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:41.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:41.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.288 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1006404377 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/1006404377", 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/1006404377", 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/1006404377", 2024-09-13T05:20:41.290 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/1006404377", 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "fd91f7e2-580d-47ea-9db5-792c8f1ebe01", 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.980804443359375, 2024-09-13T05:20:41.291 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 20, 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 29, 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 90, 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 21, 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6816", 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6817", 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6818", 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6819", 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6822", 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6823", 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6820", 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.295 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6821", 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1834661236 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6817/1834661236", 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6819/1834661236", 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6823/1834661236", 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6821/1834661236", 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "01af0f4c-5da2-46fe-a262-59fcc6a2efa9", 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.236358642578125, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 12, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 38, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 54, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 87, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 40, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:41.297 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6808", 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6809", 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:41.298 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6810", 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6811", 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:41.299 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.300 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6814", 2024-09-13T05:20:41.300 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:41.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.300 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6815", 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6812", 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6813", 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3457694380 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6809/3457694380", 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6811/3457694380", 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6815/3457694380", 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6813/3457694380", 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "64cdd5f2-1568-4e96-ad90-6ef7414651f0", 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 12, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 90, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-13T05:20:41.303 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6800", 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6801", 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6802", 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6803", 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.305 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.307 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-13T05:20:41.307 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.307 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6806", 2024-09-13T05:20:41.307 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:41.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.308 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6807", 2024-09-13T05:20:41.308 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:41.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.308 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.308 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6804", 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.79:6805", 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2778785304 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.79:6801/2778785304", 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.79:6803/2778785304", 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.79:6807/2778785304", 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.79:6805/2778785304", 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-13T05:20:41.310 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.193869+0000", 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.255764+0000", 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.286354+0000", 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.221310+0000", 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 21 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:41.313 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.026235+0000", 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 40 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138322906710015, 2024-09-13T05:20:41.314 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-13T05:20:41.315 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-13T05:18:05.010242+0000", 2024-09-13T05:20:41.315 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-13T05:20:41.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.315 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:41.315 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2024-09-13T05:20:41.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.315 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: 0, 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "3.6", 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: 3 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.316 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_primaries": [], 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2024-09-13T05:20:41.317 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2024-09-13T05:20:41.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:41.319 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:41.321 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:41.321 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:41.322 INFO:tasks.thrashosds.thrasher:j is {'epoch': 92, 'fsid': '1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5', 'created': '2024-09-13T05:17:57.363809+0000', 'modified': '2024-09-13T05:20:39.951098+0000', 'last_up_change': '2024-09-13T05:19:34.047702+0000', 'last_in_change': '2024-09-13T05:20:35.926417+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T05:18:08.122164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8695272207260132, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T05:18:15.001664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 73, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8695272207260132, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 3, 'pool_name': 'new_default_pool21435', 'create_time': '2024-09-13T05:19:21.793970+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 50, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8695272207260132, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}], 'osds': [{'osd': 0, 'uuid': 'e58174f4-85ec-4321-8bc3-384261c59c6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 77, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 2153409514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 2153409514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 2153409514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 2153409514}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 2153409514}]}, 'public_addr': '172.21.15.16:6801/2153409514', 'cluster_addr': '172.21.15.16:6803/2153409514', 'heartbeat_back_addr': '172.21.15.16:6807/2153409514', 'heartbeat_front_addr': '172.21.15.16:6805/2153409514', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '84a3db6c-dacd-43d5-b964-37fdef0091a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 2608485151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 2608485151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 2608485151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 2608485151}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 2608485151}]}, 'public_addr': '172.21.15.16:6809/2608485151', 'cluster_addr': '172.21.15.16:6811/2608485151', 'heartbeat_back_addr': '172.21.15.16:6815/2608485151', 'heartbeat_front_addr': '172.21.15.16:6813/2608485151', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0d530476-cf38-4544-9ce7-e8ed35df220b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 1006404377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 1006404377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 1006404377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 1006404377}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 1006404377}]}, 'public_addr': '172.21.15.16:6817/1006404377', 'cluster_addr': '172.21.15.16:6819/1006404377', 'heartbeat_back_addr': '172.21.15.16:6823/1006404377', 'heartbeat_front_addr': '172.21.15.16:6821/1006404377', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fd91f7e2-580d-47ea-9db5-792c8f1ebe01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.980804443359375, 'last_clean_begin': 12, 'last_clean_end': 20, 'up_from': 29, 'up_thru': 90, 'down_at': 21, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6816', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6817', 'nonce': 1834661236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6818', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6819', 'nonce': 1834661236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6822', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6823', 'nonce': 1834661236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6820', 'nonce': 1834661236}, {'type': 'v1', 'addr': '172.21.15.79:6821', 'nonce': 1834661236}]}, 'public_addr': '172.21.15.79:6817/1834661236', 'cluster_addr': '172.21.15.79:6819/1834661236', 'heartbeat_back_addr': '172.21.15.79:6823/1834661236', 'heartbeat_front_addr': '172.21.15.79:6821/1834661236', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '01af0f4c-5da2-46fe-a262-59fcc6a2efa9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.236358642578125, 'last_clean_begin': 12, 'last_clean_end': 38, 'up_from': 54, 'up_thru': 87, 'down_at': 40, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6808', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6809', 'nonce': 3457694380}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6810', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6811', 'nonce': 3457694380}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6814', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6815', 'nonce': 3457694380}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6812', 'nonce': 3457694380}, {'type': 'v1', 'addr': '172.21.15.79:6813', 'nonce': 3457694380}]}, 'public_addr': '172.21.15.79:6809/3457694380', 'cluster_addr': '172.21.15.79:6811/3457694380', 'heartbeat_back_addr': '172.21.15.79:6815/3457694380', 'heartbeat_front_addr': '172.21.15.79:6813/3457694380', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '64cdd5f2-1568-4e96-ad90-6ef7414651f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6800', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6801', 'nonce': 2778785304}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6802', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6803', 'nonce': 2778785304}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6806', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6807', 'nonce': 2778785304}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.79:6804', 'nonce': 2778785304}, {'type': 'v1', 'addr': '172.21.15.79:6805', 'nonce': 2778785304}]}, 'public_addr': '172.21.15.79:6801/2778785304', 'cluster_addr': '172.21.15.79:6803/2778785304', 'heartbeat_back_addr': '172.21.15.79:6807/2778785304', 'heartbeat_front_addr': '172.21.15.79:6805/2778785304', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.193869+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.255764+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.286354+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.221310+0000', 'dead_epoch': 21}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.026235+0000', 'dead_epoch': 40}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T05:18:05.010242+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [0, 1]}, {'pgid': '3.6', 'osds': [1, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T05:20:41.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:20:41.322 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:41.317+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:41.323 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:41.421 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:41.417+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:41.447 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:41.447 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:41.462 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:41.522 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:41.519+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:41.584 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:41.584 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:41.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:41.640 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:41.639+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:41.669 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:41.670 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:20:41.687 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-13T05:20:40.086748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149562274,"num_objects":78,"num_object_clones":20,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":9,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":12,"num_read":5847,"num_read_kb":189328,"num_write":953,"num_write_kb":275306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":75497618,"num_keys_recovered":17,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":913,"ondisk_log_size":913,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":454688,"kb_used_data":261704,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":565776352,"statfs":{"total":579820584960,"available":579354984448,"internally_reserved":0,"allocated":267984896,"data_stored":285033090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":9,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388573,"num_keys_recovered":-5,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001758"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":234,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.947557+0000","last_change":"2024-09-13T05:18:09.803701+0000","last_active":"2024-09-13T05:20:35.947557+0000","last_peered":"2024-09-13T05:20:35.947557+0000","last_clean":"2024-09-13T05:20:35.947557+0000","last_became_active":"2024-09-13T05:18:09.801680+0000","last_became_peered":"2024-09-13T05:18:09.801680+0000","last_unstale":"2024-09-13T05:20:35.947557+0000","last_undegraded":"2024-09-13T05:20:35.947557+0000","last_fullsized":"2024-09-13T05:20:35.947557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:53:37.159839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"74'302","reported_seq":1968,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:36.894682+0000","last_change":"2024-09-13T05:18:18.324779+0000","last_active":"2024-09-13T05:20:36.894682+0000","last_peered":"2024-09-13T05:20:36.894682+0000","last_clean":"2024-09-13T05:20:36.894682+0000","last_became_active":"2024-09-13T05:18:16.121457+0000","last_became_peered":"2024-09-13T05:18:16.121457+0000","last_unstale":"2024-09-13T05:20:36.894682+0000","last_undegraded":"2024-09-13T05:20:36.894682+0000","last_fullsized":"2024-09-13T05:20:36.894682+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:49:04.268070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078942599999999997,"stat_sum":{"num_bytes":16777254,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3608,"num_read_kb":23883,"num_write":312,"num_write_kb":20774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"64'5","reported_seq":2,"reported_epoch":90,"state":"peering","last_fresh":"2024-09-13T05:20:30.916050+0000","last_change":"2024-09-13T05:20:29.903561+0000","last_active":"2024-09-13T05:20:29.903561+0000","last_peered":"2024-09-13T05:19:48.449549+0000","last_clean":"2024-09-13T05:19:48.449549+0000","last_became_active":"2024-09-13T05:19:23.455354+0000","last_became_peered":"2024-09-13T05:19:23.455354+0000","last_unstale":"2024-09-13T05:20:30.916050+0000","last_undegraded":"2024-09-13T05:20:30.916050+0000","last_fullsized":"2024-09-13T05:20:30.916050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:46:53.659312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017913759999999999,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":25,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"72'64","reported_seq":355,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.939782+0000","last_change":"2024-09-13T05:18:18.140961+0000","last_active":"2024-09-13T05:20:35.939782+0000","last_peered":"2024-09-13T05:20:35.939782+0000","last_clean":"2024-09-13T05:20:35.939782+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:20:35.939782+0000","last_undegraded":"2024-09-13T05:20:35.939782+0000","last_fullsized":"2024-09-13T05:20:35.939782+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063619200000000005,"stat_sum":{"num_bytes":25165824,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":207,"num_read_kb":27196,"num_write":60,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.939841+0000","last_change":"2024-09-13T05:20:18.431968+0000","last_active":"2024-09-13T05:20:35.939841+0000","last_peered":"2024-09-13T05:20:35.939841+0000","last_clean":"2024-09-13T05:20:35.939841+0000","last_became_active":"2024-09-13T05:20:18.431262+0000","last_became_peered":"2024-09-13T05:20:18.431262+0000","last_unstale":"2024-09-13T05:20:35.939841+0000","last_undegraded":"2024-09-13T05:20:35.939841+0000","last_fullsized":"2024-09-13T05:20:35.939841+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:31:28.923680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"72'23","reported_seq":2,"reported_epoch":90,"state":"peering","last_fresh":"2024-09-13T05:20:30.911694+0000","last_change":"2024-09-13T05:20:29.911752+0000","last_active":"2024-09-13T05:20:29.911752+0000","last_peered":"2024-09-13T05:20:04.366549+0000","last_clean":"2024-09-13T05:20:04.366549+0000","last_became_active":"2024-09-13T05:18:43.272263+0000","last_became_peered":"2024-09-13T05:18:43.272263+0000","last_unstale":"2024-09-13T05:20:30.911694+0000","last_undegraded":"2024-09-13T05:20:30.911694+0000","last_fullsized":"2024-09-13T05:20:30.911694+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:11:18.106127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388662,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":64,"num_read_kb":8202,"num_write":22,"num_write_kb":16386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.947477+0000","last_change":"2024-09-13T05:19:25.538085+0000","last_active":"2024-09-13T05:20:35.947477+0000","last_peered":"2024-09-13T05:20:35.947477+0000","last_clean":"2024-09-13T05:20:35.947477+0000","last_became_active":"2024-09-13T05:19:23.462578+0000","last_became_peered":"2024-09-13T05:19:23.462578+0000","last_unstale":"2024-09-13T05:20:35.947477+0000","last_undegraded":"2024-09-13T05:20:35.947477+0000","last_fullsized":"2024-09-13T05:20:35.947477+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:24:48.193321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056272899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"73'80","reported_seq":434,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.936323+0000","last_change":"2024-09-13T05:19:24.517596+0000","last_active":"2024-09-13T05:20:35.936323+0000","last_peered":"2024-09-13T05:20:35.936323+0000","last_clean":"2024-09-13T05:20:35.936323+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:20:35.936323+0000","last_undegraded":"2024-09-13T05:20:35.936323+0000","last_fullsized":"2024-09-13T05:20:35.936323+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169184,"stat_sum":{"num_bytes":16781331,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":247,"num_read_kb":38365,"num_write":76,"num_write_kb":38938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"48'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.935962+0000","last_change":"2024-09-13T05:19:25.535626+0000","last_active":"2024-09-13T05:20:35.935962+0000","last_peered":"2024-09-13T05:20:35.935962+0000","last_clean":"2024-09-13T05:20:35.935962+0000","last_became_active":"2024-09-13T05:19:23.458938+0000","last_became_peered":"2024-09-13T05:19:23.458938+0000","last_unstale":"2024-09-13T05:20:35.935962+0000","last_undegraded":"2024-09-13T05:20:35.935962+0000","last_fullsized":"2024-09-13T05:20:35.935962+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:28:08.756172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000291315,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-13T05:20:36.956120+0000","last_change":"2024-09-13T05:20:36.956120+0000","last_active":"2024-09-13T05:20:36.956120+0000","last_peered":"2024-09-13T05:20:36.956120+0000","last_clean":"2024-09-13T05:20:36.956120+0000","last_became_active":"2024-09-13T05:20:36.955570+0000","last_became_peered":"2024-09-13T05:20:36.955570+0000","last_unstale":"2024-09-13T05:20:36.956120+0000","last_undegraded":"2024-09-13T05:20:36.956120+0000","last_fullsized":"2024-09-13T05:20:36.956120+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:40:20.616181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086185799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"72'66","reported_seq":307,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-13T05:20:37.060293+0000","last_change":"2024-09-13T05:20:37.060293+0000","last_active":"2024-09-13T05:20:37.060293+0000","last_peered":"2024-09-13T05:20:37.060293+0000","last_clean":"2024-09-13T05:20:37.060293+0000","last_became_active":"2024-09-13T05:20:36.957689+0000","last_became_peered":"2024-09-13T05:20:36.957689+0000","last_unstale":"2024-09-13T05:20:37.060293+0000","last_undegraded":"2024-09-13T05:20:37.060293+0000","last_fullsized":"2024-09-13T05:20:37.060293+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T06:29:41.023217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074498200000000004,"stat_sum":{"num_bytes":16777254,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":196,"num_read_kb":16483,"num_write":80,"num_write_kb":32804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":33554508,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"64'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.947250+0000","last_change":"2024-09-13T05:20:18.435166+0000","last_active":"2024-09-13T05:20:35.947250+0000","last_peered":"2024-09-13T05:20:35.947250+0000","last_clean":"2024-09-13T05:20:35.947250+0000","last_became_active":"2024-09-13T05:20:18.434390+0000","last_became_peered":"2024-09-13T05:20:18.434390+0000","last_unstale":"2024-09-13T05:20:35.947250+0000","last_undegraded":"2024-09-13T05:20:35.947250+0000","last_fullsized":"2024-09-13T05:20:35.947250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:49:17.104799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092053199999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"73'48","reported_seq":376,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.947611+0000","last_change":"2024-09-13T05:19:35.195473+0000","last_active":"2024-09-13T05:20:35.947611+0000","last_peered":"2024-09-13T05:20:35.947611+0000","last_clean":"2024-09-13T05:20:35.947611+0000","last_became_active":"2024-09-13T05:19:35.077291+0000","last_became_peered":"2024-09-13T05:19:35.077291+0000","last_unstale":"2024-09-13T05:20:35.947611+0000","last_undegraded":"2024-09-13T05:20:35.947611+0000","last_fullsized":"2024-09-13T05:20:35.947611+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:44:50.348812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035515500000000001,"stat_sum":{"num_bytes":25215090,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":217,"num_read_kb":19140,"num_write":46,"num_write_kb":41011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":2,"reported_epoch":90,"state":"peering","last_fresh":"2024-09-13T05:20:35.329715+0000","last_change":"2024-09-13T05:20:36.313100+0000","last_active":"2024-09-13T05:20:36.313100+0000","last_peered":"2024-09-13T05:20:35.329715+0000","last_clean":"2024-09-13T05:20:35.329715+0000","last_became_active":"2024-09-13T05:20:30.919829+0000","last_became_peered":"2024-09-13T05:20:30.919829+0000","last_unstale":"2024-09-13T05:20:35.329715+0000","last_undegraded":"2024-09-13T05:20:35.329715+0000","last_fullsized":"2024-09-13T05:20:35.329715+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:50.322912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089178300000000005,"stat_sum":{"num_bytes":51,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":51,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"72'46","reported_seq":189,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.936078+0000","last_change":"2024-09-13T05:20:18.434016+0000","last_active":"2024-09-13T05:20:35.936078+0000","last_peered":"2024-09-13T05:20:35.936078+0000","last_clean":"2024-09-13T05:20:35.936078+0000","last_became_active":"2024-09-13T05:20:18.433147+0000","last_became_peered":"2024-09-13T05:20:18.433147+0000","last_unstale":"2024-09-13T05:20:35.936078+0000","last_undegraded":"2024-09-13T05:20:35.936078+0000","last_fullsized":"2024-09-13T05:20:35.936078+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:29:49.051074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777254,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":300,"num_read_kb":12518,"num_write":32,"num_write_kb":24582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-13T05:20:36.956306+0000","last_change":"2024-09-13T05:20:36.956306+0000","last_active":"2024-09-13T05:20:36.956306+0000","last_peered":"2024-09-13T05:20:36.956306+0000","last_clean":"2024-09-13T05:20:36.956306+0000","last_became_active":"2024-09-13T05:20:36.955672+0000","last_became_peered":"2024-09-13T05:20:36.955672+0000","last_unstale":"2024-09-13T05:20:36.956306+0000","last_undegraded":"2024-09-13T05:20:36.956306+0000","last_fullsized":"2024-09-13T05:20:36.956306+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:19:22.438530+0000","last_clean_scrub_stamp":"2024-09-13T05:19:22.438530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:13:06.821319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"74'158","reported_seq":1039,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-13T05:20:35.947832+0000","last_change":"2024-09-13T05:18:18.324349+0000","last_active":"2024-09-13T05:20:35.947832+0000","last_peered":"2024-09-13T05:20:35.947832+0000","last_clean":"2024-09-13T05:20:35.947832+0000","last_became_active":"2024-09-13T05:18:16.127271+0000","last_became_peered":"2024-09-13T05:18:16.127271+0000","last_unstale":"2024-09-13T05:20:35.947832+0000","last_undegraded":"2024-09-13T05:20:35.947832+0000","last_fullsized":"2024-09-13T05:20:35.947832+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:14.356786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034066400000000002,"stat_sum":{"num_bytes":23089167,"num_objects":12,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":859,"num_read_kb":43414,"num_write":160,"num_write_kb":49237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":51,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":148971836,"num_objects":70,"num_object_clones":20,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":12,"num_read":5698,"num_read_kb":189201,"num_write":788,"num_write_kb":272884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":75497567,"num_keys_recovered":17,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":260419584,"data_stored":281379356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607587,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81908,"kb_used_data":50784,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94289932,"statfs":{"total":96636764160,"available":96552890368,"internally_reserved":0,"allocated":52002816,"data_stored":51335965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.76600000000000001,"15min":0.77600000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68300000000000005,"15min":0.68700000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0229999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.58499999999999996}]},{"osd":1,"last update":"Fri Sep 13 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.53800000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98699999999999999,"5min":1.026,"15min":1.026},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.503,"5min":0.495,"15min":0.495},"max":{"1min":0.86199999999999999,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.64200000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.89500000000000002,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.84499999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":4,"up_from":54,"seq":231928233998,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70180,"kb_used_data":42072,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94301660,"statfs":{"total":96636764160,"available":96564899840,"internally_reserved":0,"allocated":43081728,"data_stored":42406252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.66700000000000004}]},{"osd":1,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.79200000000000004}]},{"osd":2,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.76500000000000001}]},{"osd":3,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.42599999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.38100000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051608,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29124,"kb_used_data":1016,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342716,"statfs":{"total":96636764160,"available":96606941184,"internally_reserved":0,"allocated":1040384,"data_stored":403643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.75},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.85399999999999998}]},{"osd":1,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.79900000000000004}]},{"osd":2,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.90000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.55800000000000005}]}]},{"osd":2,"up_from":12,"seq":51539607586,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73188,"kb_used_data":37976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94298652,"statfs":{"total":96636764160,"available":96561819648,"internally_reserved":0,"allocated":38887424,"data_stored":38228186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.61899999999999999,"15min":0.629},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.018,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Fri Sep 13 05:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.089,"5min":1.171,"15min":1.171},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.98099999999999998,"5min":1.21,"15min":1.21},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Sep 13 05:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.86399999999999999}]},{"osd":4,"last update":"Fri Sep 13 05:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.80500000000000005}]},{"osd":5,"last update":"Fri Sep 13 05:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.872,"15min":0.88300000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.88100000000000001,"15min":0.89700000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.161,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.96699999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607586,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110732,"kb_used_data":75520,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94261108,"statfs":{"total":96636764160,"available":96523374592,"internally_reserved":0,"allocated":77332480,"data_stored":87167867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88800000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":1.0780000000000001}]},{"osd":2,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.58599999999999997,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":1.026}]},{"osd":3,"last update":"Fri Sep 13 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.90600000000000003}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.753},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.99299999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.84399999999999997,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.054,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.056}]}]},{"osd":0,"up_from":12,"seq":51539607586,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89556,"kb_used_data":54336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282284,"statfs":{"total":96636764160,"available":96545058816,"internally_reserved":0,"allocated":55640064,"data_stored":65491177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.71299999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.286,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.42299999999999999}]},{"osd":2,"last update":"Fri Sep 13 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80300000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.32600000000000001}]},{"osd":3,"last update":"Fri Sep 13 05:19:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.77700000000000002}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.63800000000000001}]},{"osd":5,"last update":"Fri Sep 13 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79300000000000004,"15min":0.79200000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0269999999999999,"5min":1.224,"15min":1.224},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.96199999999999997,"5min":1.694,"15min":1.694},"last":0.75800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54575104,"data_stored":65077320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75673600,"data_stored":86163642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37818368,"data_stored":37814310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42008576,"data_stored":41992344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":50343936,"data_stored":50331740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:20:41.689 INFO:tasks.thrashosds.thrasher:Setting 2.6 to [4, 5, 2, 1] 2024-09-13T05:20:41.689 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.6', '4', '5', '2', '1'] 2024-09-13T05:20:41.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.6 4 5 2 1 2024-09-13T05:20:41.718 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:41.718 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:41.723 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:41.721+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:41.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:41.808 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-09-13T05:20:42.234 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:42.234 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:42.234 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:42.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:20:42.236 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:42.231+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:42.237 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:42.237 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd1'\'']/id' 2024-09-13T05:20:42.237 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=24303 2024-09-13T05:20:42.237 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 24303 2024-09-13T05:20:42.237 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 24303 -C rbd-nbd 2024-09-13T05:20:42.237 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:42.238 INFO:tasks.workunit.client.0.smithi122.stdout: 24303 ? 00:00:00 rbd-nbd 2024-09-13T05:20:42.238 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:42.238 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:42.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:blkdiscard: /dev/nbd1 contains existing file system (ext2). 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:42.245 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:blkdiscard: /dev/nbd1 contains existing file system (ext2). 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:42.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-09-13T05:20:42.247 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=0 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21435 2024-09-13T05:20:42.248 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-13T05:20:42.249 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:42.084+0000 7f2531ffb640 -1 JournalMetadata: 0x55693eb99cc0 failed to locate client: INSPECT 2024-09-13T05:20:42.249 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=4 2024-09-13T05:20:42.249 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 0 -lt 10 ']' 2024-09-13T05:20:42.249 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -gt 0 ']' 2024-09-13T05:20:42.249 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd21435 --verbose 2024-09-13T05:20:42.249 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=1 2024-09-13T05:20:42.249 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.925142+0000" 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=2 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:42.250 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.941026+0000" 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=3 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.978228+0000" 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=4 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:42.251 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:42.252 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:42.016811+0000" 2024-09-13T05:20:42.252 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:42.252 INFO:tasks.workunit.client.0.smithi122.stdout:Summary: 2024-09-13T05:20:42.252 INFO:tasks.workunit.client.0.smithi122.stdout: 4 entries inspected, 0 errors 2024-09-13T05:20:42.252 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:42.164+0000 7f8f811d7640 -1 JournalMetadata: 0x555e8b4b25b0 failed to locate client: INSPECT 2024-09-13T05:20:42.252 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 1 2024-09-13T05:20:42.337 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:42.335+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:42.412 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:42.412 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:42.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:42.478 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:42.477+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:42.540 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:42.537+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:42.544 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:42.544 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:42.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:42.646 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:42.645+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:42.721 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:42.721 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:42.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:42.778 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:42.777+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:42.842 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:42.841+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:42.905 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:42.905 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:42.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:42.943 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:42.939+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.6 pg_upmap_items mapping to [4->5,2->1] 2024-09-13T05:20:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:20:43.044 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:43.043+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:43.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:43.098 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:43.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:43.145 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:43.143+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:43.173 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21435 2024-09-13T05:20:43.173 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-13T05:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:43.241 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:43.245 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:43.241+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:43.259 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:43.259 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:43.275 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:43.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:43.346 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:43.345+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.117257851000005, 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.106052090000006, 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.095873036, 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.088437232999993, 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.081829529999993, 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.072603311999998, 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.457529815999999, 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003044452, 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987382+0000", 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988524+0000", 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011418859999999999 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988545+0000", 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0483e-05 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988556+0000", 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1586999999999999e-05 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988662+0000", 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106137 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988680+0000", 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7295999999999998e-05 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.990428+0000", 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017486789999999999 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.447598474999999, 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002020534, 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998323+0000", 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010079329999999999 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998341+0000", 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8077e-05 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998351+0000", 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0570999999999999e-05 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998429+0000", 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7350999999999999e-05 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998439+0000", 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0651e-05 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.999336+0000", 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000896469 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.443158107, 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021888099999999998, 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001755+0000", 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002748+0000", 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099270400000000007 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002765+0000", 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7218e-05 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002776+0000", 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0783e-05 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002832+0000", 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6181999999999998e-05 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002842+0000", 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5950000000000005e-06 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.003944+0000", 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011029569999999999 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.437853763, 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00206362, 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007059+0000", 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008052+0000", 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099299999999999996 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008070+0000", 2024-09-13T05:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7742999999999999e-05 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008081+0000", 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0920999999999999e-05 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008137+0000", 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5773000000000003e-05 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008146+0000", 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6469999999999993e-06 2024-09-13T05:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.009124+0000", 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097727500000000011 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.42884452, 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024079549999999998, 2024-09-13T05:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016068+0000", 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017401+0000", 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013326989999999999 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017440+0000", 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9623000000000003e-05 2024-09-13T05:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017469+0000", 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8776e-05 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017538+0000", 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9375999999999994e-05 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017548+0000", 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8050000000000001e-06 2024-09-13T05:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.018477+0000", 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00092904299999999995 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.423764973000001, 2024-09-13T05:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030823669999999999, 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021147+0000", 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022523+0000", 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013751779999999999 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022563+0000", 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0009999999999998e-05 2024-09-13T05:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022591+0000", 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8683e-05 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022701+0000", 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110173 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022721+0000", 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9911e-05 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.024231+0000", 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001509886 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.414710219, 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031121460000000001, 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030202+0000", 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031555+0000", 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001352518 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031595+0000", 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0197000000000002e-05 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031646+0000", 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0742000000000003e-05 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031787+0000", 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014145999999999999 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031806+0000", 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8845e-05 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.033316+0000", 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015096739999999999 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.40849873, 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025441959999999999, 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036414+0000", 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037751+0000", 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013372289999999999 2024-09-13T05:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037788+0000", 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7023000000000001e-05 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037814+0000", 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6308999999999998e-05 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037903+0000", 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8930000000000004e-05 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037915+0000", 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1474000000000001e-05 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.038959+0000", 2024-09-13T05:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010445610000000001 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.403002429000001, 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025004459999999999, 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041910+0000", 2024-09-13T05:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043262+0000", 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001352645 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043304+0000", 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1746000000000003e-05 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043335+0000", 2024-09-13T05:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0618999999999999e-05 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043438+0000", 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000103261 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043451+0000", 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.261e-05 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.044412+0000", 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00096126200000000005 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.397192118, 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038240729999999999, 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047720+0000", 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049066+0000", 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013460589999999999 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049110+0000", 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3702999999999998e-05 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049137+0000", 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7443000000000001e-05 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049277+0000", 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013948399999999999 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049298+0000", 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1234e-05 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.051546+0000", 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022478400000000001 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.386649643, 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025842819999999998, 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058262+0000", 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059597+0000", 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001334256 2024-09-13T05:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059635+0000", 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8562000000000001e-05 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059662+0000", 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6703000000000002e-05 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059751+0000", 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9163e-05 2024-09-13T05:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059762+0000", 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0671e-05 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.060848+0000", 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001086697 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.380880836999999, 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030803480000000001, 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064032+0000", 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065378+0000", 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013465630000000001 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065418+0000", 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0422e-05 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065446+0000", 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7477000000000001e-05 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065577+0000", 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131254 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065599+0000", 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1656999999999999e-05 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.067113+0000", 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001514409 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.374618719000001, 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030880719999999999, 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070294+0000", 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071663+0000", 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001369089 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071704+0000", 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0745999999999999e-05 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071731+0000", 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7182000000000002e-05 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071865+0000", 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013386800000000001 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071885+0000", 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0800000000000001e-05 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.073383+0000", 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001497757 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.368270505, 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033570940000000001, 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076642+0000", 2024-09-13T05:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077619+0000", 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097751200000000004 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077660+0000", 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0368999999999999e-05 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077690+0000", 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0139999999999999e-05 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077823+0000", 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013348400000000001 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077842+0000", 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8255999999999999e-05 2024-09-13T05:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.080000+0000", 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021586169999999998 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:43.537 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:43.538 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:43.533+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:43.638 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:43.635+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:43.668 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:43.669 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:43.669 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:43.669 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:43.683 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:43.768 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:43.767+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:43.840 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:43.835+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:43.843 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:43.843 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:43.843 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:43.843 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:43.843 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:43.857 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:43.944 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:43.943+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:44.000 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:43.996+0000 7f6fbf7fe640 -1 JournalMetadata: 0x55b46b5871a0 failed to locate client: INSPECT 2024-09-13T05:20:44.011 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=4 2024-09-13T05:20:44.011 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=1 2024-09-13T05:20:44.011 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1 -lt 10 ']' 2024-09-13T05:20:44.011 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -gt 0 ']' 2024-09-13T05:20:44.011 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd21435 --verbose 2024-09-13T05:20:44.037 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:44.037 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:44.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:44.037 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.90211299500001, 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:20:44.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.90148173599999, 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:20:44.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.900836615, 2024-09-13T05:20:44.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:20:44.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.050 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.90032925200001, 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.054 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.899803816, 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:20:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.899166304, 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.059 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:20:44.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:20:44.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.89858959599999, 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:20:44.063 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.89803812299999, 2024-09-13T05:20:44.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:44.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.89757753999999, 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.89706765400001, 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.079 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.89664059099999, 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 153.50988087299999, 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.084 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 152.89234606100001, 2024-09-13T05:20:44.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:20:44.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:20:44.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.661567125999994, 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:20:44.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:20:44.095 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:20:44.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.650946782000005, 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.097 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.098 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.099 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:20:44.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.102 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.643732467999996, 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:44.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:20:44.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:20:44.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:20:44.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:20:44.107 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.637376947999996, 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:20:44.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:20:44.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:20:44.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.637320583999994, 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:20:44.113 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:44.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:20:44.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:20:44.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:20:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.637283013000001, 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:44.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:20:44.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:20:44.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.123 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.637248489000001, 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:20:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:20:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.129 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.129 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:44.130 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=1 2024-09-13T05:20:44.130 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:44.130 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.925142+0000" 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=2 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.941026+0000" 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=3 2024-09-13T05:20:44.131 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.978228+0000" 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=4 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:42.016811+0000" 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:44.132 INFO:tasks.workunit.client.0.smithi122.stdout:Summary: 2024-09-13T05:20:44.133 INFO:tasks.workunit.client.0.smithi122.stdout: 4 entries inspected, 0 errors 2024-09-13T05:20:44.133 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:44.096+0000 7fe68a7fc640 -1 JournalMetadata: 0x562e06fb6b20 failed to locate client: INSPECT 2024-09-13T05:20:44.133 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 1 2024-09-13T05:20:44.133 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:44.134 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:44.127+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:44.231 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:44.228+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:44.303 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:44.303 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:44.303 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:44.303 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:44.317 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:44.332 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:44.328+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:44.433 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:44.431+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:44.439 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:44.439 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:44.439 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:44.439 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:44.439 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:44.453 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:44.533 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:44.528+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:44.625 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:44.625 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:44.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:44.625 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:44.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.625 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:44.625 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:44.625 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.281658371000006, 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:20:44.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:20:44.628 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.269777769000001, 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.631 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.632 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:44.635 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.263834070000001, 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.639 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.261158362000003, 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:20:44.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:20:44.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.256444329000004, 2024-09-13T05:20:44.644 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.240492055999994, 2024-09-13T05:20:44.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:20:44.652 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.230851853999994, 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:44.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:20:44.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:20:44.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.223759048999995, 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:20:44.660 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.215541092999999, 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.661 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.662 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.21339528, 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:20:44.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.211477486000007, 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:20:44.669 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:20:44.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.058031224000004, 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:20:44.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:20:44.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.051593284000006, 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:44.678 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.680 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:301 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 921600~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.045525131000005, 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001711162, 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.682 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 301 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576987+0000", 2024-09-13T05:20:44.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576986+0000", 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.576993+0000", 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.3499999999999999e-06 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.684 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577011+0000", 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7802e-05 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577031+0000", 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9585999999999998e-05 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577862+0000", 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00083072400000000002 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.577982+0000", 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120104 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.578699+0000", 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00071695299999999999 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:44.686 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.037981578, 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.688 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:20:44.690 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:312 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.025304332999994, 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016207330000000001, 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 312 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.691 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597208+0000", 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597207+0000", 2024-09-13T05:20:44.692 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597214+0000", 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5849999999999996e-06 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597228+0000", 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4487e-05 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.597237+0000", 2024-09-13T05:20:44.693 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2660000000000007e-06 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598030+0000", 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00079273199999999999 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598143+0000", 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011332000000000001 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.598829+0000", 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00068562600000000005 2024-09-13T05:20:44.694 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 68.538680467000006, 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:44.695 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.696 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.697 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.698 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.699 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 68.538611646000007, 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.700 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:44.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:44.702 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:20:44.703 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 64.510209075999995, 2024-09-13T05:20:44.704 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.706 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.707 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:20:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:44.709 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.710 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 55.983528761999999, 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.711 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:44.712 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.713 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.714 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:44.715 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:44.716 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:44.718 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:44.718 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:44.716+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:44.819 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:44.815+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:44.845 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:44.845 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:44.845 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:44.845 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:44.860 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:44.919 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:44.916+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:45.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:20:45.028 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:45.028 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:45.028 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:45.028 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:45.028 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:45.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:45.058 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:45.056+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:45.107 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21435 2024-09-13T05:20:45.108 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-13T05:20:45.121 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:45.119+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:45.165 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.119386516000006, 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:45.166 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:45.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:45.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.170 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.082985937999993, 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:45.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:45.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.174 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.079402634000004, 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.074768547999994, 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.071531272000001, 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:45.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:45.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.067851872999995, 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.190 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.064495679999993, 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:45.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:45.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.058164387000005, 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:45.194 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:45.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.050484437999998, 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.198 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:45.201 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.047853294000006, 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.202 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.044011334000004, 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.040175938000004, 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.210 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.213 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.036720692000003, 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:45.217 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.032646851999999, 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.218 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.029567923000002, 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.221 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:45.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:45.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:45.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.025324906999998, 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.225 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.227 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.228 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.021448905, 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.231 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.232 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.013823279999997, 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:45.233 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.234 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:45.235 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.010556870000002, 2024-09-13T05:20:45.236 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.237 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.239 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 70.006864467, 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:45.240 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:45.241 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:45.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:45.244 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:45.244 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.244 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.244 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.244 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:45.244 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:45.244 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:45.246 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:45.172+0000 7efe84a0d640 -1 JournalMetadata: 0x55767c0b7870 failed to locate client: INSPECT 2024-09-13T05:20:45.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=4 2024-09-13T05:20:45.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=2 2024-09-13T05:20:45.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 2 -lt 10 ']' 2024-09-13T05:20:45.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -gt 0 ']' 2024-09-13T05:20:45.246 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd21435 --verbose 2024-09-13T05:20:45.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:45.247 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=1 2024-09-13T05:20:45.247 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:45.247 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:45.247 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-09-13T05:20:45.247 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:45.247 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.925142+0000" 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=2 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.941026+0000" 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=3 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-09-13T05:20:45.248 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.978228+0000" 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=4 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:42.016811+0000" 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout:Summary: 2024-09-13T05:20:45.249 INFO:tasks.workunit.client.0.smithi122.stdout: 4 entries inspected, 0 errors 2024-09-13T05:20:45.250 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:45.240+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:45.250 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:45.244+0000 7faaddffb640 -1 JournalMetadata: 0x560a5988d3b0 failed to locate client: INSPECT 2024-09-13T05:20:45.254 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 1 2024-09-13T05:20:45.346 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:45.343+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:45.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:45.374 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:45.374 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:45.374 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:45.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:45.447 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:45.443+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:45.513 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:45.531 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:45.532 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:45.548 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:45.546+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:45.549 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:45.549 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:45.570 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:45.571 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:45.574 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:45.574 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:45.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:45.592 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:45.592 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:45.611 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:45.611 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:45.674 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:45.672+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:45.710 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:45.710 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:45.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:45.766 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:45.764+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:45.851 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:45.846+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:45.893 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:45.894 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:45.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:45.954 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:45.952+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:45.999 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:20:46.028 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:46.029 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:46.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:46.086 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:46.084+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:46.154 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:46.152+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:46.209 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:46.210 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:46.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:46.255 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:46.250+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:46.256 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21435 2024-09-13T05:20:46.256 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-13T05:20:46.340 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:46.336+0000 7f641b7fe640 -1 JournalMetadata: 0x5584653b3eb0 failed to locate client: INSPECT 2024-09-13T05:20:46.346 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=4 2024-09-13T05:20:46.346 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=3 2024-09-13T05:20:46.346 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 3 -lt 10 ']' 2024-09-13T05:20:46.346 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -gt 0 ']' 2024-09-13T05:20:46.346 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd21435 --verbose 2024-09-13T05:20:46.356 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:46.354+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:46.394 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:46.394 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:46.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:46.419 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=1 2024-09-13T05:20:46.419 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:46.419 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:46.419 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-09-13T05:20:46.419 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:46.419 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:46.419 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.925142+0000" 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=2 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 0, 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.941026+0000" 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=3 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:46.420 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:41.978228+0000" 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout:Entry: tag_id=1, commit_tid=4 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout:{ 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout: "event_type": "AioDiscard", 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout: "offset": 4096, 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout: "length": 4096, 2024-09-13T05:20:46.421 INFO:tasks.workunit.client.0.smithi122.stdout: "discard_granularity_bytes": 4096, 2024-09-13T05:20:46.422 INFO:tasks.workunit.client.0.smithi122.stdout: "timestamp": "2024-09-13T05:20:42.016811+0000" 2024-09-13T05:20:46.422 INFO:tasks.workunit.client.0.smithi122.stdout:} 2024-09-13T05:20:46.422 INFO:tasks.workunit.client.0.smithi122.stdout:Summary: 2024-09-13T05:20:46.422 INFO:tasks.workunit.client.0.smithi122.stdout: 4 entries inspected, 0 errors 2024-09-13T05:20:46.423 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:46.420+0000 7fd054f63640 -1 JournalMetadata: 0x556cc32d3b90 failed to locate client: INSPECT 2024-09-13T05:20:46.428 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 1 2024-09-13T05:20:46.456 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:46.454+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:46.535 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:46.535 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:46.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:46.594 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:46.592+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:46.657 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:46.652+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:46.674 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:46.674 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:46.689 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:46.758 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:46.756+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:46.819 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:46.819 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:46.833 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:46.880 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:46.878+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:46.954 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:46.954 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:46.960 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:46.958+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:46.969 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:47.061 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:47.060+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:47.137 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:47.138 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:47.152 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:47.196 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:47.194+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:47.263 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:47.258+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:47.275 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:47.275 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:47.289 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:47.365 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:47.360+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:47.431 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21435 2024-09-13T05:20:47.431 INFO:tasks.workunit.client.0.smithi122.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-13T05:20:47.453 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:47.453 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:47.466 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:47.464+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:47.468 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:47.518 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:47.516+0000 7f8c377fe640 -1 JournalMetadata: 0x562cd2f3b280 failed to locate client: INSPECT 2024-09-13T05:20:47.525 INFO:tasks.workunit.client.0.smithi122.stderr:+ queue_length=0 2024-09-13T05:20:47.525 INFO:tasks.workunit.client.0.smithi122.stderr:+ tries=4 2024-09-13T05:20:47.525 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 4 -lt 10 ']' 2024-09-13T05:20:47.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 0 -gt 0 ']' 2024-09-13T05:20:47.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 0 -eq 0 ']' 2024-09-13T05:20:47.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd1 24303 2024-09-13T05:20:47.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd1 2024-09-13T05:20:47.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=24303 2024-09-13T05:20:47.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-13T05:20:47.526 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:47.526 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:47.527 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:47.527 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:47.528 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:47.528 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:47.528 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-13T05:20:47.569 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:47.566+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:47.584 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:47.584 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:47.598 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:47.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:47.633 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:20:47.633 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^24303\b' 2024-09-13T05:20:47.633 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^24303\b' 2024-09-13T05:20:47.684 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:47.682+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:47.690 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:47.691 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:20:47.691 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *24303\b' 2024-09-13T05:20:47.691 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *24303\b' 2024-09-13T05:20:47.697 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:47.697 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:47.762 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:47.762 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:47.774 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:47.772+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:47.776 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:47.790 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:47.791 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:47.807 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:47.807 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:47.823 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:47.841 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:47.841 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:47.860 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:47.861 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:47.875 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:47.870+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:47.881 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:47.937 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:47.937 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:47.940 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:47.940 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:47.941 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:47.951 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:47.992 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:47.990+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:48.002 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:20:48.002 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:20:48.002 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T05:20:48.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-09-13T05:20:48.024 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:20:48.098 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:48.096+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:48.120 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:48.120 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:48.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:48.120 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:48.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.120 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:48.120 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.071797885999999, 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:48.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:48.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:48.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:48.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.123 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.123 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:48.123 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:48.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.123 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.124 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:48.125 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:48.126 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.035397308, 2024-09-13T05:20:48.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:48.127 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.127 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.127 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.127 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.127 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.127 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:48.127 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:48.128 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.129 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:48.130 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:48.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.131 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.131 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.131 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:48.131 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:48.131 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.131 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.031814003999997, 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.132 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.134 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.135 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.027179918000002, 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.138 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.139 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:48.140 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.023942641999994, 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:48.141 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.142 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.143 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:48.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.020263243000002, 2024-09-13T05:20:48.145 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.146 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.147 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:48.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:48.149 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.01690705, 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:48.150 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.151 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.152 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:48.153 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.010575756999998, 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.154 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:48.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:48.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.157 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.157 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:48.157 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:48.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.158 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.002895808000005, 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.159 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.160 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:48.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:48.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.000264663999999, 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.164 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.165 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.166 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.996422703999997, 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.167 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.169 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.170 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.992587307999997, 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.171 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.172 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:48.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:48.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:48.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:48.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.175 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.989132061999996, 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.179 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.985058222000006, 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:48.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:48.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:48.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:48.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:48.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.184 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.981979292999995, 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:48.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:48.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.977736277000005, 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.192 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:48.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:48.192 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.192 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:48.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:48.192 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.973860275000007, 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.193 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:48.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.966234650000004, 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.197 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.198 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.200 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.962968239999995, 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.201 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:48.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:48.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:48.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:48.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:48.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.205 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 72.959275837000007, 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:48.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:48.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:48.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:48.209 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:48.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:48.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:48.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:48.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:48.209 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:48.210 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.210 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.210 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.210 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:48.210 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:48.210 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:48.211 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:20:48.211 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd feature disable rbd/testrbdnbd21435 journaling 2024-09-13T05:20:48.212 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:48.206+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:48.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:48.284 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config image rm rbd/testrbdnbd21435 rbd_discard_granularity_bytes 2024-09-13T05:20:48.312 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:48.308+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:48.319 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-09-13T05:20:48.314+0000 7f9b566ea640 -1 mgr.server reply reply (2) No such file or directory no change 2024-09-13T05:20:48.319 INFO:teuthology.orchestra.run.smithi016.stdout:moved 2 / 34 (5.88235%) 2024-09-13T05:20:48.319 INFO:teuthology.orchestra.run.smithi016.stdout:avg 5.66667 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 1.24722 -> 1.24722 (expected baseline 2.17307) 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.0 with 4 -> 4 pgs (0.705882 -> 0.705882 * mean) 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.2 with 7 -> 7 pgs (1.23529 -> 1.23529 * mean) 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:oload 110 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 1 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0008 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0009 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 weight 1.0000 -> 0.7215 2024-09-13T05:20:48.320 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 weight 1.0000 -> 0.8922 2024-09-13T05:20:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:no change 2024-09-13T05:20:48.338 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:48.338 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:48.341 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-13T05:20:48.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-09-13T05:20:48.356 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:48.358 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:20:48.358 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:20:48.359 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:20:48.359 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:20:48.360 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:20:48.361 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:20:48.361 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:20:48.362 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:20:48.413 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:48.410+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:48.514 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:48.510+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:48.516 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:48.516 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:48.516 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:48.516 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:48.516 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:48.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:48.615 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:48.612+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:48.653 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:48.653 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:48.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:48.653 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:48.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.653 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:20:48.653 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:48.653 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 151.51649897300001, 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:20:48.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:20:48.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 150.50671744300001, 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.660 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:48.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:20:48.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:20:48.666 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.283887190000002, 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.670 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.276857594999996, 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.671 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:20:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:48.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.236074759999994, 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:20:48.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:20:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.229216287, 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:20:48.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 85.197074326000006, 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.583112851999999, 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:48.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:20:48.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:20:48.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.582951625999996, 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:48.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:20:48.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:20:48.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:80 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'83, mlcod=57'80)", 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.734214567999999, 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072315670000000004, 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915606+0000", 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:48.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915894+0000", 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028823899999999998 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915934+0000", 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9659000000000001e-05 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915953+0000", 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8660000000000001e-05 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916091+0000", 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790800000000001 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916150+0000", 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9079000000000001e-05 2024-09-13T05:20:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922698+0000", 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065481339999999997 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922839+0000", 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140805 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.708 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:81 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'84, mlcod=57'80)", 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.729004074000002, 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003098127, 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920815+0000", 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920950+0000", 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502499999999999 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920989+0000", 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8779000000000003e-05 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921013+0000", 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3753000000000001e-05 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921422+0000", 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040899799999999999 2024-09-13T05:20:48.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921481+0000", 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8811999999999999e-05 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923815+0000", 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002333852 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923916+0000", 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101032 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:169 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'92, mlcod=65'89)", 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.319395489000001, 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004357662, 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330424+0000", 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330542+0000", 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117475 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330570+0000", 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8036999999999999e-05 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330585+0000", 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4983e-05 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330651+0000", 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6361999999999999e-05 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330674+0000", 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3064000000000001e-05 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334660+0000", 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039862229999999997 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334784+0000", 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012329499999999999 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:170 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'93, mlcod=65'89)", 2024-09-13T05:20:48.717 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.314198841, 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035268080000000002, 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335621+0000", 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335688+0000", 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7006000000000002e-05 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335716+0000", 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.798e-05 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335728+0000", 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1712999999999999e-05 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335838+0000", 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110575 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335878+0000", 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9969000000000003e-05 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339094+0000", 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032155399999999998 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339149+0000", 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5685e-05 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.721 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:172 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'95, mlcod=65'89)", 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.312058237999999, 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029153970000000001, 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337763+0000", 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337762+0000", 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337854+0000", 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2342999999999993e-05 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337868+0000", 2024-09-13T05:20:48.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3658e-05 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.337876+0000", 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5900000000000002e-06 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338010+0000", 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013445199999999999 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.338063+0000", 2024-09-13T05:20:48.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2762000000000001e-05 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340641+0000", 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002577663 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.340679+0000", 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8172e-05 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:238 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'101, mlcod=66'99)", 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.200728220000002, 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048419240000000001, 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:48.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449091+0000", 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450611+0000", 2024-09-13T05:20:48.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015204629999999999 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450655+0000", 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3662999999999999e-05 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450676+0000", 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0573000000000001e-05 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450825+0000", 2024-09-13T05:20:48.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014922399999999999 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450859+0000", 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3779999999999998e-05 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453864+0000", 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030051959999999999 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453935+0000", 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1302000000000003e-05 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:323 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'103, mlcod=66'100)", 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.272577976999997, 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029606749999999999, 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377243+0000", 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377314+0000", 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1248999999999995e-05 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377333+0000", 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924999999999999e-05 2024-09-13T05:20:48.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377344+0000", 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0817e-05 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377395+0000", 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0803000000000001e-05 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377416+0000", 2024-09-13T05:20:48.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0993e-05 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380171+0000", 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027549660000000002 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380204+0000", 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3331999999999999e-05 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5559.0:252 2.7 e73/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 73'147, mlcod=72'144)", 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.252499516, 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037496309999999998, 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397329+0000", 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:48.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397341+0000", 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1321000000000001e-05 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397349+0000", 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2490000000000001e-06 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397396+0000", 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7172999999999997e-05 2024-09-13T05:20:48.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397420+0000", 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3828999999999998e-05 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401029+0000", 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003609396 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401072+0000", 2024-09-13T05:20:48.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2308999999999998e-05 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:258 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 3194880~999424] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.238270501000002, 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047026610000000003, 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:48.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 258 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411550+0000", 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412658+0000", 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001108314 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412672+0000", 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3682999999999999e-05 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412678+0000", 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1739999999999997e-06 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412716+0000", 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7290999999999999e-05 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412768+0000", 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2710999999999997e-05 2024-09-13T05:20:48.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412844+0000", 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6157999999999998e-05 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.414915+0000", 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020706159999999999 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:48.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416189+0000", 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012737110000000001 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416226+0000", 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7082000000000002e-05 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416254+0000", 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7840000000000001e-05 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:260 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 28672~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.217144322999999, 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027961330000000001, 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:48.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 260 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432677+0000", 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432676+0000", 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432689+0000", 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3546999999999999e-05 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432704+0000", 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4508000000000001e-05 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432710+0000", 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9920000000000004e-06 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432774+0000", 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4343000000000004e-05 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432880+0000", 2024-09-13T05:20:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106202 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433059+0000", 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017896599999999999 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434279+0000", 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012192660000000001 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:48.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435285+0000", 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001006142 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435415+0000", 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130314 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435473+0000", 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8357999999999999e-05 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:262 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 2150400~4096] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.217083465000002, 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028404789999999999, 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 262 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.751 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432738+0000", 2024-09-13T05:20:48.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432743+0000", 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.434e-06 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432749+0000", 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7899999999999996e-06 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.432752+0000", 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.552e-06 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433532+0000", 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00077913799999999997 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433639+0000", 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107366 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.433812+0000", 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000172585 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.434709+0000", 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089765699999999995 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435516+0000", 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080721099999999995 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435554+0000", 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7672000000000001e-05 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.435579+0000", 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4491999999999999e-05 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:48.756 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:48.758 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-09-13T05:20:48.758 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:48.758 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:48.758 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:48.759 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd1'\'']/id' 2024-09-13T05:20:48.759 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:48.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:48.759 INFO:teuthology.orchestra.run.smithi016.stdout:moved 0 / 34 (0%) 2024-09-13T05:20:48.759 INFO:teuthology.orchestra.run.smithi016.stdout:avg 5.66667 2024-09-13T05:20:48.759 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 1.24722 -> 1.24722 (expected baseline 2.17307) 2024-09-13T05:20:48.759 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.0 with 4 -> 4 pgs (0.705882 -> 0.705882 * mean) 2024-09-13T05:20:48.760 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.2 with 7 -> 7 pgs (1.23529 -> 1.23529 * mean) 2024-09-13T05:20:48.760 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:48.760 INFO:teuthology.orchestra.run.smithi016.stdout:oload 1000 2024-09-13T05:20:48.760 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 1 2024-09-13T05:20:48.760 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-13T05:20:48.760 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0008 2024-09-13T05:20:48.760 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0085 2024-09-13T05:20:48.761 INFO:teuthology.orchestra.run.smithi016.stderr:no change 2024-09-13T05:20:48.761 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T05:20:48.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 1.0 2024-09-13T05:20:48.761 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:48.756+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:48.814 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=24740 2024-09-13T05:20:48.814 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 24740 2024-09-13T05:20:48.814 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 24740 -C rbd-nbd 2024-09-13T05:20:48.820 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:48.820 INFO:tasks.workunit.client.0.smithi122.stdout: 24740 ? 00:00:00 rbd-nbd 2024-09-13T05:20:48.820 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd feature enable rbd/testrbdnbd21435 journaling 2024-09-13T05:20:48.858 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:48.856+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:48.896 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-13T05:20:48.897 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:48.897 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:48.898 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:48.898 INFO:tasks.workunit.client.0.smithi122.stderr:++ which blkdiscard 2024-09-13T05:20:48.899 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-13T05:20:48.900 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:48.900 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-13T05:20:48.931 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:48.932 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:48.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:48.990 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:48.988+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:20:49.074 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:49.072+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:49.105 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:49.106 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:49.107 INFO:teuthology.orchestra.run.smithi016.stdout:moved 0 / 34 (0%) 2024-09-13T05:20:49.107 INFO:teuthology.orchestra.run.smithi016.stdout:avg 5.66667 2024-09-13T05:20:49.107 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 1.24722 -> 1.24722 (expected baseline 2.17307) 2024-09-13T05:20:49.107 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.0 with 4 -> 4 pgs (0.705882 -> 0.705882 * mean) 2024-09-13T05:20:49.108 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.2 with 7 -> 7 pgs (1.23529 -> 1.23529 * mean) 2024-09-13T05:20:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:49.108 INFO:teuthology.orchestra.run.smithi016.stdout:oload 1000 2024-09-13T05:20:49.108 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 1 2024-09-13T05:20:49.108 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-13T05:20:49.108 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0008 2024-09-13T05:20:49.108 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0085 2024-09-13T05:20:49.108 INFO:teuthology.orchestra.run.smithi016.stderr:no change 2024-09-13T05:20:49.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:49.126 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T05:20:49.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-09-13T05:20:49.130 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd feature disable rbd/testrbdnbd21435 journaling 2024-09-13T05:20:49.162 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:49.158+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:49.211 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd1 24740 2024-09-13T05:20:49.211 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd1 2024-09-13T05:20:49.211 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=24740 2024-09-13T05:20:49.211 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-13T05:20:49.211 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:49.212 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:49.212 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:49.213 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:49.214 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:49.214 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:49.215 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-13T05:20:49.262 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:49.260+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:49.285 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 158.768115091, 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:20:49.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:20:49.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:20:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 158.76784306600001, 2024-09-13T05:20:49.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:20:49.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:20:49.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:20:49.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 158.76779948000001, 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:49.297 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:20:49.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:20:49.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 158.76776725299999, 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:20:49.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:20:49.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.308 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 158.76755610500001, 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.309 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:49.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:20:49.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:20:49.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 158.76751981199999, 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:20:49.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:20:49.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:20:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.319 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 158.76748136200001, 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.320 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:20:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:20:49.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:20:49.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 158.76730645999999, 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:49.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:20:49.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:20:49.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:20:49.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.330 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.952602522000007, 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:49.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:20:49.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:20:49.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:20:49.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:20:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.945072409000005, 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:20:49.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:20:49.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:20:49.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.340 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.935503081999997, 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:20:49.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.345 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.924340209999997, 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.346 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:20:49.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.918180613000004, 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:20:49.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:20:49.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:20:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.357 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.869344833, 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.362 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 87.861068298999996, 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:49.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:20:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:20:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:20:49.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.368307891000001, 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:20:49.367 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.372 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.367872744000003, 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:49.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:20:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.378 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.367558752000001, 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.367153500999997, 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:20:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:20:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:20:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.366828963000003, 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:20:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:20:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:20:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.394 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:49.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:49.396 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:49.392+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:49.397 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:20:49.397 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^24740\b' 2024-09-13T05:20:49.397 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^24740\b' 2024-09-13T05:20:49.397 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:49.397 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:20:49.397 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *24740\b' 2024-09-13T05:20:49.397 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *24740\b' 2024-09-13T05:20:49.397 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:49.397 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:49.496 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:49.492+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:49.564 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:49.564 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:49.564 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:49.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:49.622 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:49.620+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:49.698 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:49.694+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:49.748 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:49.748 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:49.748 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:49.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:49.806 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:49.804+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:49.884 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.553026719000002, 2024-09-13T05:20:49.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:49.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.541820958000002, 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.890 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:49.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:49.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.531641903999997, 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:49.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.524206101000004, 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:49.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:49.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:49.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:49.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.517598398000004, 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:49.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:49.909 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.508372179999995, 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:49.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.893298684000001, 2024-09-13T05:20:49.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003044452, 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987382+0000", 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988524+0000", 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011418859999999999 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988545+0000", 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0483e-05 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988556+0000", 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1586999999999999e-05 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988662+0000", 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106137 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988680+0000", 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7295999999999998e-05 2024-09-13T05:20:49.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.990428+0000", 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017486789999999999 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.883367343, 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002020534, 2024-09-13T05:20:49.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998323+0000", 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010079329999999999 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998341+0000", 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8077e-05 2024-09-13T05:20:49.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998351+0000", 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0570999999999999e-05 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998429+0000", 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7350999999999999e-05 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998439+0000", 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0651e-05 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.999336+0000", 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000896469 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.878926974999999, 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021888099999999998, 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001755+0000", 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002748+0000", 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099270400000000007 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002765+0000", 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7218e-05 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002776+0000", 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0783e-05 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002832+0000", 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6181999999999998e-05 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002842+0000", 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5950000000000005e-06 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.003944+0000", 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011029569999999999 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:49.926 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.873622631, 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00206362, 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007059+0000", 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008052+0000", 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099299999999999996 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008070+0000", 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7742999999999999e-05 2024-09-13T05:20:49.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008081+0000", 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0920999999999999e-05 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008137+0000", 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5773000000000003e-05 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008146+0000", 2024-09-13T05:20:49.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6469999999999993e-06 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.009124+0000", 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097727500000000011 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.864613387999999, 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024079549999999998, 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016068+0000", 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017401+0000", 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013326989999999999 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017440+0000", 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9623000000000003e-05 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017469+0000", 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8776e-05 2024-09-13T05:20:49.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017538+0000", 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9375999999999994e-05 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017548+0000", 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8050000000000001e-06 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.018477+0000", 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00092904299999999995 2024-09-13T05:20:49.933 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.859533841000001, 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030823669999999999, 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021147+0000", 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022523+0000", 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013751779999999999 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022563+0000", 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0009999999999998e-05 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022591+0000", 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8683e-05 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022701+0000", 2024-09-13T05:20:49.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110173 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022721+0000", 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9911e-05 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.024231+0000", 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001509886 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.937 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.850479087, 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031121460000000001, 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030202+0000", 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031555+0000", 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001352518 2024-09-13T05:20:49.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031595+0000", 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0197000000000002e-05 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031646+0000", 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0742000000000003e-05 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031787+0000", 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014145999999999999 2024-09-13T05:20:49.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031806+0000", 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8845e-05 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.033316+0000", 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015096739999999999 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.844267597999998, 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025441959999999999, 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:49.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036414+0000", 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037751+0000", 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013372289999999999 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037788+0000", 2024-09-13T05:20:49.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7023000000000001e-05 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037814+0000", 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6308999999999998e-05 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037903+0000", 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8930000000000004e-05 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037915+0000", 2024-09-13T05:20:49.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1474000000000001e-05 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.038959+0000", 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010445610000000001 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.838771297000001, 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025004459999999999, 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.945 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041910+0000", 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043262+0000", 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001352645 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043304+0000", 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1746000000000003e-05 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043335+0000", 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0618999999999999e-05 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043438+0000", 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000103261 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043451+0000", 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.261e-05 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.044412+0000", 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00096126200000000005 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.832960986, 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038240729999999999, 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.949 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047720+0000", 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049066+0000", 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013460589999999999 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049110+0000", 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3702999999999998e-05 2024-09-13T05:20:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049137+0000", 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7443000000000001e-05 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049277+0000", 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013948399999999999 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049298+0000", 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1234e-05 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.051546+0000", 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022478400000000001 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.822418510999999, 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025842819999999998, 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058262+0000", 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059597+0000", 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001334256 2024-09-13T05:20:49.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059635+0000", 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8562000000000001e-05 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059662+0000", 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6703000000000002e-05 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059751+0000", 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9163e-05 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059762+0000", 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0671e-05 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.060848+0000", 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001086697 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.957 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.816649705, 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030803480000000001, 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064032+0000", 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065378+0000", 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013465630000000001 2024-09-13T05:20:49.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065418+0000", 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0422e-05 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065446+0000", 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7477000000000001e-05 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065577+0000", 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131254 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065599+0000", 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1656999999999999e-05 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.067113+0000", 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001514409 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.961 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.810387587000001, 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030880719999999999, 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070294+0000", 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071663+0000", 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001369089 2024-09-13T05:20:49.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071704+0000", 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0745999999999999e-05 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071731+0000", 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7182000000000002e-05 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071865+0000", 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013386800000000001 2024-09-13T05:20:49.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071885+0000", 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0800000000000001e-05 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.073383+0000", 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001497757 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:49.965 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.804039372999998, 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033570940000000001, 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076642+0000", 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077619+0000", 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097751200000000004 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077660+0000", 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0368999999999999e-05 2024-09-13T05:20:49.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077690+0000", 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0139999999999999e-05 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077823+0000", 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013348400000000001 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077842+0000", 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8255999999999999e-05 2024-09-13T05:20:49.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.080000+0000", 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021586169999999998 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:49.969 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:49.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:49.971 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:49.972 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:20:49.972 INFO:tasks.workunit.client.0.smithi122.stderr:++ ceph-conf --show-config-value librados_thread_count 2024-09-13T05:20:49.972 INFO:tasks.workunit.client.0.smithi122.stderr:+ EXPECTED=2 2024-09-13T05:20:49.973 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:20:49.973 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:20:49.973 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:20:49.973 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:20:49.973 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:20:49.973 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:20:49.973 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:20:49.973 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21435 2024-09-13T05:20:49.973 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:49.970+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:50.016 INFO:teuthology.orchestra.run.smithi016.stdout:moved 2 / 34 (5.88235%) 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:avg 5.66667 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 1.24722 -> 1.24722 (expected baseline 2.17307) 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.0 with 4 -> 4 pgs (0.705882 -> 0.705882 * mean) 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.2 with 7 -> 7 pgs (1.23529 -> 1.23529 * mean) 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:oload 110 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 3 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0008 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0009 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 weight 1.0000 -> 0.7216 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 weight 1.0000 -> 0.8127 2024-09-13T05:20:50.017 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 weight 1.0000 -> 0.8921 2024-09-13T05:20:50.046 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T05:20:50.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2024-09-13T05:20:50.090 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:50.088+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:50.106 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:50.106 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:50.106 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:50.106 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:50.120 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:50.170 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:50.170 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:50.172 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:50.168+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:50.212 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:50.212 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:50.232 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:50.232 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:50.243 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:50.243 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:50.243 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:50.243 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:50.243 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:50.251 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:50.257 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:50.269 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:50.269 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:50.273 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:50.268+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:50.374 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:50.372+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:50.377 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:50.377 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:50.391 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:50.474 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:50.470+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:50.512 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:50.512 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:50.526 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:50.616 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:50.614+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:50.677 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:50.674+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:50.690 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:50.690 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:50.704 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:50.792 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:50.790+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:50.869 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:50.869 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:50.878 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:50.876+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:50.883 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:50.979 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:50.974+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:moved 3 / 34 (8.82353%) 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:avg 5.66667 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 1.24722 -> 2.0548 (expected baseline 2.17307) 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.0 with 4 -> 5 pgs (0.705882 -> 0.882353 * mean) 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.2 with 7 -> 9 pgs (1.23529 -> 1.58824 * mean) 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:oload 110 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 1 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0008 2024-09-13T05:20:51.027 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0009 2024-09-13T05:20:51.028 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 weight 0.7216 -> 0.5207 2024-09-13T05:20:51.028 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 weight 0.8127 -> 0.6605 2024-09-13T05:20:51.028 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 weight 0.8921 -> 0.7959 2024-09-13T05:20:51.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T05:20:51.051 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:51.051 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:51.065 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:51.108 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:51.106+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:51.180 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:51.176+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:51.228 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:51.228 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:51.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:51.281 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:51.276+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:51.366 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:51.366 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:51.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:51.381 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:51.380+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:51.483 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:51.478+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:51.505 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:51.505 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:51.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:51.584 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:51.580+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:51.637 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:51.638 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:51.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:51.694 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:51.692+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:51.786 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:51.784+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:51.821 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:51.821 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:51.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:51.926 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:51.924+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:51.988 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:51.982+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:52.010 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:52.010 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:52.025 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:52.028 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-13T05:20:52.098 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:52.096+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:52.166 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:52.167 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:52.178 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-09-13T05:20:52.178 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:52.178 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:52.179 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:52.179 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:52.179 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd1'\'']/id' 2024-09-13T05:20:52.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:52.230 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:52.228+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:52.261 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=24852 2024-09-13T05:20:52.261 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 24852 2024-09-13T05:20:52.262 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 24852 -C rbd-nbd 2024-09-13T05:20:52.268 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:52.268 INFO:tasks.workunit.client.0.smithi122.stdout: 24852 ? 00:00:00 rbd-nbd 2024-09-13T05:20:52.269 INFO:tasks.workunit.client.0.smithi122.stderr:++ ps -p 24852 -T 2024-09-13T05:20:52.269 INFO:tasks.workunit.client.0.smithi122.stderr:++ grep -c io_context_pool 2024-09-13T05:20:52.276 INFO:tasks.workunit.client.0.smithi122.stderr:+ ACTUAL=2 2024-09-13T05:20:52.276 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 2 -eq 2 ']' 2024-09-13T05:20:52.276 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd1 24852 2024-09-13T05:20:52.276 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd1 2024-09-13T05:20:52.276 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=24852 2024-09-13T05:20:52.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-13T05:20:52.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:52.277 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:52.277 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:52.277 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:52.278 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:52.278 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:52.278 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-13T05:20:52.290 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:52.286+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:52.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:52.363 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:52.363 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:52.379 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:20:52.379 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^24852\b' 2024-09-13T05:20:52.379 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^24852\b' 2024-09-13T05:20:52.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:52.407 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:52.404+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:52.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:52.440 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:52.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:20:52.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *24852\b' 2024-09-13T05:20:52.441 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *24852\b' 2024-09-13T05:20:52.447 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:52.448 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:52.451 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:52.467 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:52.468 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:52.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:52.488 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:52.499 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:52.494+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:52.508 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:52.508 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:52.511 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:52.511 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:52.511 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:52.511 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:52.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:52.528 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:52.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:52.604+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:52.696 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:52.694+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:52.735 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:52.735 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:52.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:52.795 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:52.790+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:52.871 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.539973919999994, 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:20:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:20:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.528768158999995, 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:20:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:20:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:20:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:20:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:20:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:20:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.518589105000004, 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:20:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.511153301999997, 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:20:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:20:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:20:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.504545598999997, 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:20:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.495319381000002, 2024-09-13T05:20:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:20:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:20:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.880245885000001, 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003044452, 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987382+0000", 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988524+0000", 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011418859999999999 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988545+0000", 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0483e-05 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988556+0000", 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1586999999999999e-05 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988662+0000", 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106137 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988680+0000", 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7295999999999998e-05 2024-09-13T05:20:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.990428+0000", 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017486789999999999 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.870314543999999, 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002020534, 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.997315+0000", 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998323+0000", 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010079329999999999 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998341+0000", 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8077e-05 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998351+0000", 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0570999999999999e-05 2024-09-13T05:20:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998429+0000", 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7350999999999999e-05 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.998439+0000", 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0651e-05 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.999336+0000", 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000896469 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.865874175999998, 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021888099999999998, 2024-09-13T05:20:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001756+0000", 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.001755+0000", 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002748+0000", 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099270400000000007 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002765+0000", 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7218e-05 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002776+0000", 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0783e-05 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002832+0000", 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6181999999999998e-05 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.002842+0000", 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5950000000000005e-06 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.003944+0000", 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011029569999999999 2024-09-13T05:20:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.860569831999999, 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00206362, 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007060+0000", 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.007059+0000", 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008052+0000", 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00099299999999999996 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008070+0000", 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7742999999999999e-05 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008081+0000", 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0920999999999999e-05 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008137+0000", 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5773000000000003e-05 2024-09-13T05:20:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.008146+0000", 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6469999999999993e-06 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.009124+0000", 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097727500000000011 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.851560589000002, 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024079549999999998, 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016069+0000", 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.016068+0000", 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017401+0000", 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013326989999999999 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017440+0000", 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9623000000000003e-05 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017469+0000", 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8776e-05 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017538+0000", 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9375999999999994e-05 2024-09-13T05:20:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.017548+0000", 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8050000000000001e-06 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.018477+0000", 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00092904299999999995 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.846481042000001, 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030823669999999999, 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021147+0000", 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022523+0000", 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013751779999999999 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022563+0000", 2024-09-13T05:20:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0009999999999998e-05 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022591+0000", 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8683e-05 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022701+0000", 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110173 2024-09-13T05:20:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022721+0000", 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9911e-05 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.024231+0000", 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001509886 2024-09-13T05:20:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.837426288, 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031121460000000001, 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030202+0000", 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031555+0000", 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001352518 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031595+0000", 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0197000000000002e-05 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031646+0000", 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0742000000000003e-05 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031787+0000", 2024-09-13T05:20:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014145999999999999 2024-09-13T05:20:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031806+0000", 2024-09-13T05:20:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8845e-05 2024-09-13T05:20:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.033316+0000", 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015096739999999999 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.831214799000001, 2024-09-13T05:20:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025441959999999999, 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036415+0000", 2024-09-13T05:20:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.036414+0000", 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037751+0000", 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013372289999999999 2024-09-13T05:20:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037788+0000", 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7023000000000001e-05 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037814+0000", 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6308999999999998e-05 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037903+0000", 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8930000000000004e-05 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.037915+0000", 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1474000000000001e-05 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.038959+0000", 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010445610000000001 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.825718498000001, 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025004459999999999, 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041911+0000", 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.041910+0000", 2024-09-13T05:20:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043262+0000", 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001352645 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043304+0000", 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1746000000000003e-05 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043335+0000", 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0618999999999999e-05 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043438+0000", 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000103261 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.043451+0000", 2024-09-13T05:20:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.261e-05 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.044412+0000", 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00096126200000000005 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.819908186999999, 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038240729999999999, 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047720+0000", 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049066+0000", 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013460589999999999 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049110+0000", 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3702999999999998e-05 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049137+0000", 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7443000000000001e-05 2024-09-13T05:20:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049277+0000", 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013948399999999999 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049298+0000", 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1234e-05 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.051546+0000", 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022478400000000001 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.809365712000002, 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025842819999999998, 2024-09-13T05:20:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058264+0000", 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.058262+0000", 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059597+0000", 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001334256 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059635+0000", 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8562000000000001e-05 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059662+0000", 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6703000000000002e-05 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059751+0000", 2024-09-13T05:20:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9163e-05 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.059762+0000", 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0671e-05 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.060848+0000", 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001086697 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.803596905999999, 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030803480000000001, 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064032+0000", 2024-09-13T05:20:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065378+0000", 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013465630000000001 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065418+0000", 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0422e-05 2024-09-13T05:20:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065446+0000", 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7477000000000001e-05 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065577+0000", 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131254 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065599+0000", 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1656999999999999e-05 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.067113+0000", 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001514409 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.797334788000001, 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030880719999999999, 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070294+0000", 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071663+0000", 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001369089 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071704+0000", 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0745999999999999e-05 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071731+0000", 2024-09-13T05:20:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7182000000000002e-05 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071865+0000", 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013386800000000001 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071885+0000", 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0800000000000001e-05 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.073383+0000", 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001497757 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.790986574000001, 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033570940000000001, 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076642+0000", 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077619+0000", 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097751200000000004 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077660+0000", 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0368999999999999e-05 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077690+0000", 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0139999999999999e-05 2024-09-13T05:20:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077823+0000", 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013348400000000001 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077842+0000", 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8255999999999999e-05 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.080000+0000", 2024-09-13T05:20:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021586169999999998 2024-09-13T05:20:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:52.980 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:52.983 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:52.983 INFO:tasks.workunit.client.0.smithi122.stderr:+ EXPECTED=7 2024-09-13T05:20:52.983 INFO:tasks.workunit.client.0.smithi122.stderr:++ _sudo rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd21435 2024-09-13T05:20:52.983 INFO:tasks.workunit.client.0.smithi122.stderr:++ local cmd 2024-09-13T05:20:52.984 INFO:tasks.workunit.client.0.smithi122.stderr:+++ id -u 2024-09-13T05:20:52.984 INFO:tasks.workunit.client.0.smithi122.stderr:++ '[' 1000 -eq 0 ']' 2024-09-13T05:20:52.984 INFO:tasks.workunit.client.0.smithi122.stderr:+++ which rbd 2024-09-13T05:20:52.984 INFO:tasks.workunit.client.0.smithi122.stderr:++ cmd=/usr/bin/rbd 2024-09-13T05:20:52.984 INFO:tasks.workunit.client.0.smithi122.stderr:++ shift 2024-09-13T05:20:52.984 INFO:tasks.workunit.client.0.smithi122.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd21435 2024-09-13T05:20:52.984 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:52.982+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:53.083 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:53.078+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:53.118 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:53.118 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:53.118 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:53.118 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:53.132 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:53.221 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:53.218+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:53.285 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:53.282+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:53.301 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:53.301 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:53.301 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:53.301 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:53.301 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:53.315 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:53.386 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:53.384+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:53.484 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:53.484 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:53.484 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:53.484 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:53.484 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.484 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:20:53.484 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.348843813, 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.485 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.486 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.487 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.488 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.34821255400001, 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.489 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.490 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:20:53.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.492 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.493 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.34756743299999, 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.498 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:20:53.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:20:53.500 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.34706007, 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.501 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.503 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.504 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:20:53.505 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.34653463399999, 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.506 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:20:53.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.508 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:20:53.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.345897122, 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:20:53.510 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.512 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.513 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.514 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.34532041400001, 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:20:53.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:112 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'74, mlcod=14'65)", 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.34476894100001, 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053859850000000003, 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.519 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136601+0000", 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136600+0000", 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136715+0000", 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011508199999999999 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136755+0000", 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0487e-05 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136765+0000", 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0759999999999992e-06 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138322+0000", 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015569620000000001 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138354+0000", 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2815999999999999e-05 2024-09-13T05:20:53.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141897+0000", 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003542922 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141987+0000", 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9530000000000005e-05 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.34430835800001, 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.343798472, 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:20:53.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:20:53.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'65)", 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 163.34337140900001, 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052543099999999999, 2024-09-13T05:20:53.532 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137998+0000", 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137997+0000", 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138132+0000", 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013500900000000001 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138151+0000", 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8924e-05 2024-09-13T05:20:53.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.138172+0000", 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140594+0000", 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024213849999999999 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140624+0000", 2024-09-13T05:20:53.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9779999999999999e-05 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143210+0000", 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0025862099999999998 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143253+0000", 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2793000000000002e-05 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:134 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'92, mlcod=14'87)", 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 162.95661169100001, 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054827549999999997, 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524758+0000", 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524748+0000", 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999905 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524829+0000", 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0897999999999997e-05 2024-09-13T05:20:53.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524839+0000", 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0047999999999999e-05 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524845+0000", 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9000000000000003e-06 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524881+0000", 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5802e-05 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.524893+0000", 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1996e-05 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530212+0000", 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053185100000000003 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.530241+0000", 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9247999999999998e-05 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:155 1.0 e15/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'109, mlcod=14'105)", 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 162.339076879, 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053052990000000003, 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142293+0000", 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142292+0000", 2024-09-13T05:20:53.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142453+0000", 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000161652 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142481+0000", 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7813e-05 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142500+0000", 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8896999999999999e-05 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142563+0000", 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2463000000000001e-05 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.142592+0000", 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9232999999999999e-05 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147536+0000", 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049440389999999999 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:11.147598+0000", 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2382e-05 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.108297944, 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:20:53.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.549 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.097677599999997, 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:53.550 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:20:53.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:53.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:20:53.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:20:53.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:20:53.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:20:53.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:53.556 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.090463286000002, 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:53.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:20:53.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:20:53.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.084107766000002, 2024-09-13T05:20:53.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:20:53.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:20:53.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.084051402, 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:20:53.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:20:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.084013830999993, 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:53.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.083979307000007, 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:53.580 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:20:53.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:53.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:20:53.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:20:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:53.586 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:53.586 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:53.587 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:53.588 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:53.586+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:53.688 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:53.684+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:53.756 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:53.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:53.757 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:53.757 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:53.771 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:53.789 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:53.784+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:53.890 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:53.888+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:53.890 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:53.890 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:53.890 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:53.891 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:53.891 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:53.906 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:53.990 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:53.986+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:54.026 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.683706736000005, 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.027 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:54.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:20:54.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.671826134, 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:20:54.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:20:54.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:54.037 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.665882435, 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:20:54.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:43 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'15, mlcod=46'12)", 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.663206727000002, 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017402139999999999, 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361354+0000", 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361352+0000", 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361368+0000", 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5109000000000001e-05 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361395+0000", 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7640999999999998e-05 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361413+0000", 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7558999999999999e-05 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361528+0000", 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000114836 2024-09-13T05:20:54.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.361576+0000", 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7954000000000002e-05 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363005+0000", 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001429427 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.363094+0000", 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9441e-05 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.658492694000003, 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:20:54.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.642540421000007, 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:20:54.054 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.632900219000007, 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:54.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:54.059 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.625807413999993, 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:20:54.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:20:54.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.063 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.617589457999998, 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:54.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.615443644999999, 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.068 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:54.072 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 92.613525851000006, 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:20:54.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:20:54.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.075 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:20:54.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:20:54.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.075 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.460079589000003, 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.077 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:20:54.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.453641649000005, 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.084 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.085 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 86.440029942999999, 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:20:54.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:54.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:20:54.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:20:54.089 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.940728832000005, 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:20:54.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:54.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:20:54.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.093 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.940660011000006, 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:20:54.094 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.095 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.097 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 73.912257440999994, 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.099 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:20:54.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:20:54.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.105 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 65.385577127000005, 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:20:54.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5603.0:28 2.1 2:96923c38:::journal_data.2.11d2ef348a4.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e95)", 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:48.266800+0000", 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.7577614009999998, 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027118490000000001, 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:54.112 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5603", 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/389233024", 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 28 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266800+0000", 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.113 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266800+0000", 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266797+0000", 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266811+0000", 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3876999999999999e-05 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266840+0000", 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8909000000000001e-05 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266860+0000", 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0089999999999999e-05 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266999+0000", 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013951999999999999 2024-09-13T05:20:54.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.267232+0000", 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000232434 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.267506+0000", 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000274228 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:54.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.268630+0000", 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0011239150000000001 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.269382+0000", 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00075244899999999998 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.269458+0000", 2024-09-13T05:20:54.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5984999999999998e-05 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.269511+0000", 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3189000000000002e-05 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5618.0:164 2.1 2:96923c38:::journal_data.2.11d2ef348a4.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e96)", 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:49.195437+0000", 2024-09-13T05:20:54.118 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 4.8291244180000001, 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0030533930000000002, 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5618", 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/919220950", 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 164 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195437+0000", 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195437+0000", 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195434+0000", 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195448+0000", 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3816e-05 2024-09-13T05:20:54.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195480+0000", 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2243000000000003e-05 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195498+0000", 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7995999999999998e-05 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195645+0000", 2024-09-13T05:20:54.121 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00014696699999999999 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195777+0000", 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013206399999999999 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.198490+0000", 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027129599999999999 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.122 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.123 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:54.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:20:54.125 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:54.125 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:54.120+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:54.225 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:54.222+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:54.295 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:54.295 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:54.295 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:54.295 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:54.309 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:54.326 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:54.324+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:54.372 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV=/dev/nbd1 2024-09-13T05:20:54.372 INFO:tasks.workunit.client.0.smithi122.stderr:+ get_pid rbd 2024-09-13T05:20:54.372 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pool=rbd 2024-09-13T05:20:54.373 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns= 2024-09-13T05:20:54.373 INFO:tasks.workunit.client.0.smithi122.stderr:++ rbd device --device-type nbd --format xml list 2024-09-13T05:20:54.373 INFO:tasks.workunit.client.0.smithi122.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21435'\''][device='\''/dev/nbd1'\'']/id' 2024-09-13T05:20:54.426 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:54.424+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:54.457 INFO:tasks.workunit.client.0.smithi122.stderr:+ PID=24917 2024-09-13T05:20:54.457 INFO:tasks.workunit.client.0.smithi122.stderr:+ test -n 24917 2024-09-13T05:20:54.457 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -p 24917 -C rbd-nbd 2024-09-13T05:20:54.462 INFO:tasks.workunit.client.0.smithi122.stdout: PID TTY TIME CMD 2024-09-13T05:20:54.462 INFO:tasks.workunit.client.0.smithi122.stdout: 24917 ? 00:00:00 rbd-nbd 2024-09-13T05:20:54.463 INFO:tasks.workunit.client.0.smithi122.stderr:++ ps -p 24917 -T 2024-09-13T05:20:54.463 INFO:tasks.workunit.client.0.smithi122.stderr:++ grep -c io_context_pool 2024-09-13T05:20:54.469 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:54.469 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:54.470 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:54.470 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:54.470 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:54.471 INFO:tasks.workunit.client.0.smithi122.stderr:+ ACTUAL=7 2024-09-13T05:20:54.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 7 -eq 7 ']' 2024-09-13T05:20:54.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ unmap_device /dev/nbd1 24917 2024-09-13T05:20:54.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ local args=/dev/nbd1 2024-09-13T05:20:54.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ local pid=24917 2024-09-13T05:20:54.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-13T05:20:54.472 INFO:tasks.workunit.client.0.smithi122.stderr:+ local cmd 2024-09-13T05:20:54.472 INFO:tasks.workunit.client.0.smithi122.stderr:++ id -u 2024-09-13T05:20:54.473 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' 1000 -eq 0 ']' 2024-09-13T05:20:54.473 INFO:tasks.workunit.client.0.smithi122.stderr:++ which rbd 2024-09-13T05:20:54.473 INFO:tasks.workunit.client.0.smithi122.stderr:+ cmd=/usr/bin/rbd 2024-09-13T05:20:54.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ shift 2024-09-13T05:20:54.474 INFO:tasks.workunit.client.0.smithi122.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-13T05:20:54.485 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:54.532 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:54.530+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:54.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:54.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd device --device-type nbd list 2024-09-13T05:20:54.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^24917\b' 2024-09-13T05:20:54.570 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^24917\b' 2024-09-13T05:20:54.611 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:54.611 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.563585144000001, 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:54.612 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.613 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.614 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.615 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.616 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.527184566000003, 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.617 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.618 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.619 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:54.620 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.523601262, 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:54.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.623 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:54.624 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.518967176000004, 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:54.625 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.626 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.627 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.628 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.515729899999997, 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:54.629 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:54.631 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.512050501000004, 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:54.632 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.633 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.634 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.635 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.508694308000003, 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.636 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:54.638 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.639 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.502363015, 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.640 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:54.641 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.642 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:54.643 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:54.644 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.494683065999993, 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.645 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:54.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.648 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.492051922000002, 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:54.649 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:54.650 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:54.651 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.488209961999999, 2024-09-13T05:20:54.652 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.654 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.655 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.656 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.484374566, 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.657 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.658 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:54.659 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.660 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.480919319999998, 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.661 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.662 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:54.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.664 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.476845479999994, 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:54.665 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:54.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:54.667 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.473766550999997, 2024-09-13T05:20:54.668 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.669 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.670 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.671 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.469523534999993, 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.672 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:54.673 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.674 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.675 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.465647532999995, 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.676 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:54.677 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.678 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.679 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.680 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.458021908000006, 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038360970000000001, 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150958+0000", 2024-09-13T05:20:54.681 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.150956+0000", 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152677+0000", 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017215640000000001 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152713+0000", 2024-09-13T05:20:54.682 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5892000000000001e-05 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152742+0000", 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8869000000000001e-05 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152935+0000", 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019264799999999999 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.683 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.152976+0000", 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1028000000000001e-05 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154794+0000", 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001818079 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:54.684 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.454755497999997, 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.686 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.687 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 79.451063094999995, 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.688 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:54.689 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:54.690 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:54.691 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:54.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:54.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ ps -C rbd-nbd 2024-09-13T05:20:54.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ expect_false grep '^ *24917\b' 2024-09-13T05:20:54.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep '^ *24917\b' 2024-09-13T05:20:54.693 INFO:tasks.workunit.client.0.smithi122.stderr:+ return 0 2024-09-13T05:20:54.694 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:54.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:54.694 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:54.688+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:54.720 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:54.720 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:54.734 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:54.734 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:54.750 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:54.750 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:54.770 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:54.790 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:54.790 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:54.794 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:54.790+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:54.810 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:54.810 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:54.810 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:54.827 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:54.918 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:54.916+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:54.995 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:54.994+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:54.996 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:54.996 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:55.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:20:55.134 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:55.132+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:55.135 INFO:tasks.workunit.client.0.smithi122.stderr:+ DEV= 2024-09-13T05:20:55.135 INFO:tasks.workunit.client.0.smithi122.stderr:+ echo OK 2024-09-13T05:20:55.135 INFO:tasks.workunit.client.0.smithi122.stdout:OK 2024-09-13T05:20:55.136 INFO:tasks.workunit.client.0.smithi122.stderr:+ cleanup 2024-09-13T05:20:55.136 INFO:tasks.workunit.client.0.smithi122.stderr:+ local ns s 2024-09-13T05:20:55.136 INFO:tasks.workunit.client.0.smithi122.stderr:+ set +e 2024-09-13T05:20:55.136 INFO:tasks.workunit.client.0.smithi122.stderr:+ mount 2024-09-13T05:20:55.136 INFO:tasks.workunit.client.0.smithi122.stderr:+ fgrep /tmp/tmp.WvgaeitFGU/mnt 2024-09-13T05:20:55.148 INFO:tasks.workunit.client.0.smithi122.stderr:+ rm -Rf /tmp/tmp.WvgaeitFGU 2024-09-13T05:20:55.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ '[' -n '' ']' 2024-09-13T05:20:55.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ for ns in '' ${NS} 2024-09-13T05:20:55.161 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace '' status testrbdnbd21435 2024-09-13T05:20:55.186 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:55.186 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:55.197 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:55.194+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:55.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:55.217 INFO:tasks.workunit.client.0.smithi122.stdout:Watchers: none 2024-09-13T05:20:55.221 INFO:tasks.workunit.client.0.smithi122.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-13T05:20:55.221 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:55.298 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:55.296+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:55.372 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:55.372 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:55.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:55.399 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:55.394+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:55.500 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:55.498+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:55.557 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:55.558 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:55.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:55.622 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:55.620+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:55.698 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:55.698 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:55.701 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:55.696+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:55.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:55.722 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace '' status testrbdnbd21435 2024-09-13T05:20:55.723 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep 'Watchers: none' 2024-09-13T05:20:55.785 INFO:tasks.workunit.client.0.smithi122.stdout:Watchers: none 2024-09-13T05:20:55.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ break 2024-09-13T05:20:55.786 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace '' snap purge testrbdnbd21435 2024-09-13T05:20:55.802 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:55.798+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:55.888 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:55.888 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:55.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:55.903 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:55.898+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:56.003 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:56.002+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:56.029 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:56.029 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:56.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:56.055 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 5, 3, 4] 2024-09-13T05:20:56.055 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T05:20:56.055 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T05:20:56.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-13T05:20:56.105 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:56.102+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:56.170 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:56.170 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:56.186 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:56.232 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:56.230+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:56.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-13T05:20:56.306 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:56.304+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:56.318 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:56.318 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:56.332 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:56.406 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:56.404+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:56.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-13T05:20:56.497 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:56.497 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:56.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:56.504+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:56.512 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:56.608 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:56.606+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:56.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-13T05:20:56.632 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:56.632 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:56.646 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:20:56.732 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:56.730+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:56.806 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:56.806 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:56.810 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:56.806+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:56.820 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:20:56.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-13T05:20:56.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:20:56.911 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:56.906+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:56.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-13T05:20:56.986 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:56.987 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:57.001 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:20:57.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:57.009 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:57.025 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:57.025 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:57.041 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:57.042 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:57.044 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:57.042+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:57.060 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:57.060 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:57.084 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:57.084 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:57.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:20:57.102 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:57.113 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:57.108+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:57.172 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:57.172 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:57.172 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:57.172 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:57.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.172 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:57.172 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.830504699000002, 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:20:57.174 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:20:57.175 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:57.176 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:41 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'13, mlcod=46'11)", 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.818624096999997, 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017442410000000001, 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.178 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352735+0000", 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.179 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352733+0000", 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352748+0000", 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5046e-05 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352776+0000", 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8688e-05 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.180 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352789+0000", 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3029e-05 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352893+0000", 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000103861 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.181 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352951+0000", 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7873e-05 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354413+0000", 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014620449999999999 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.354479+0000", 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.5806e-05 2024-09-13T05:20:57.182 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.812680397999998, 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:20:57.184 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.185 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:20:57.186 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.805290657, 2024-09-13T05:20:57.187 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:20:57.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:20:57.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.789338384000004, 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.192 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.193 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.194 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.195 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.779698182000004, 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.196 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.197 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:20:57.199 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.772605377000005, 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.200 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:20:57.201 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:20:57.202 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.203 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.204 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.764387420999995, 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.205 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:20:57.206 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:20:57.207 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.208 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.762241607999997, 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.209 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:20:57.210 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:20:57.211 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.212 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.760323814000003, 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.213 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:20:57.215 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.216 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.606877552, 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.217 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:57.218 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:20:57.219 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:57.221 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.600439612000002, 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:20:57.223 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:20:57.225 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 89.586827905999996, 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:20:57.226 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.227 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.228 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.229 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 81.087526795000002, 2024-09-13T05:20:57.230 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.231 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.232 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:20:57.233 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.234 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 81.087457974000003, 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.235 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:20:57.236 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.237 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:57.238 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.239 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 77.059055404000006, 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.240 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.241 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:20:57.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:20:57.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:20:57.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.245 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 68.532375090000002, 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.246 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:20:57.247 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.248 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.249 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.250 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.251 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5603.0:28 2.1 2:96923c38:::journal_data.2.11d2ef348a4.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e95)", 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:48.266800+0000", 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 8.9045593640000007, 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027118490000000001, 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.252 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5603", 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/389233024", 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 28 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266800+0000", 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266800+0000", 2024-09-13T05:20:57.253 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266797+0000", 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266811+0000", 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3876999999999999e-05 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.254 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266840+0000", 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8909000000000001e-05 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266860+0000", 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0089999999999999e-05 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266999+0000", 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013951999999999999 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.255 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.267232+0000", 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000232434 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.267506+0000", 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000274228 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.268630+0000", 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0011239150000000001 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.256 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.269382+0000", 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00075244899999999998 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.269458+0000", 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5984999999999998e-05 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.269511+0000", 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3189000000000002e-05 2024-09-13T05:20:57.257 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5618.0:164 2.1 2:96923c38:::journal_data.2.11d2ef348a4.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e96)", 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:49.195437+0000", 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 7.9759223810000002, 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0030533930000000002, 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5618", 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/919220950", 2024-09-13T05:20:57.258 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 164 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195437+0000", 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195437+0000", 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.259 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195434+0000", 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195448+0000", 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3816e-05 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195480+0000", 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2243000000000003e-05 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.260 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195498+0000", 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7995999999999998e-05 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195645+0000", 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00014696699999999999 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195777+0000", 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013206399999999999 2024-09-13T05:20:57.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.198490+0000", 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027129599999999999 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.5.0:73 2.4 e104/90 2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7 v 104'75, mlcod=103'71)", 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:57.072793+0000", 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.098566298999999996, 2024-09-13T05:20:57.262 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017468270000000001, 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.072793+0000", 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.072793+0000", 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.263 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.072792+0000", 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.072799+0000", 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8700000000000003e-06 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.072822+0000", 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3091999999999998e-05 2024-09-13T05:20:57.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.072834+0000", 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2172e-05 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.072944+0000", 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000109755 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.265 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.072971+0000", 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6562000000000001e-05 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.074487+0000", 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0015167329999999999 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.074539+0000", 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.1999000000000001e-05 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.266 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.267 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.267 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:57.268 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:20:57.269 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:57.266+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:57.370 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:57.366+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:57.437 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:57.438 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:57.438 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:20:57.438 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:57.452 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:20:57.496 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:57.494+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:57.571 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:57.566+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:57.619 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:57.620 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:20:57.620 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:20:57.620 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:20:57.620 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:57.634 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:20:57.671 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:57.668+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:57.772 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:57.766+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:57.796 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:20:57.796 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:20:57.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:20:57.796 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:20:57.796 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.796 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:20:57.796 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:57.796 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.748049543999997, 2024-09-13T05:20:57.796 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.797 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.798 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:20:57.799 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.800 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.711648965999998, 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.801 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.802 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:20:57.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:20:57.804 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.708065661999996, 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.805 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.806 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.807 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:20:57.808 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.703431576, 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.809 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:20:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.700194300000007, 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.815 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.816 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.696514901, 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041814690000000002, 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096929+0000", 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.096928+0000", 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098403+0000", 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001474848 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.818 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098455+0000", 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2000999999999999e-05 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098495+0000", 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9962000000000001e-05 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098719+0000", 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022375100000000001 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.819 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098759+0000", 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0417999999999998e-05 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.101111+0000", 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023518020000000001 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.820 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.693158707999999, 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040078930000000002, 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100285+0000", 2024-09-13T05:20:57.821 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.100283+0000", 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102035+0000", 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017513100000000001 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102071+0000", 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5895000000000001e-05 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102100+0000", 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8999000000000001e-05 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102240+0000", 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013983100000000001 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.102273+0000", 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3735000000000001e-05 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.104293+0000", 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020201490000000002 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.824 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.686827414999996, 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042895980000000004, 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106617+0000", 2024-09-13T05:20:57.825 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.106615+0000", 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108585+0000", 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019700550000000001 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108625+0000", 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.0027999999999997e-05 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.826 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108655+0000", 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0246000000000001e-05 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108775+0000", 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000119335 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.108812+0000", 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7447999999999998e-05 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.827 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.110906+0000", 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002094303 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.679147466000003, 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041027390000000002, 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114297+0000", 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.114294+0000", 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:57.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116413+0000", 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021189619999999998 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116456+0000", 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2302000000000003e-05 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116485+0000", 2024-09-13T05:20:57.830 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8932e-05 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116608+0000", 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012316399999999999 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116630+0000", 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2629e-05 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118399+0000", 2024-09-13T05:20:57.831 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0017691499999999999 2024-09-13T05:20:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.832 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.676516321999998, 2024-09-13T05:20:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:20:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.833 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:20:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.834 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.835 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:20:57.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:57.837 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.672674361999995, 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:20:57.838 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:20:57.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:20:57.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.668838965999996, 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:20:57.841 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.842 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.843 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:20:57.844 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.845 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.665383719999994, 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0039811179999999996, 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128060+0000", 2024-09-13T05:20:57.846 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.128059+0000", 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129945+0000", 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018861209999999999 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.847 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129985+0000", 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9654999999999999e-05 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130008+0000", 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3662000000000001e-05 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130171+0000", 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016282100000000001 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.848 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.130198+0000", 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7038000000000001e-05 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132042+0000", 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001843454 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.849 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.661309880000005, 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0040616910000000001, 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.850 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132134+0000", 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.132133+0000", 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134080+0000", 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019472960000000001 2024-09-13T05:20:57.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134115+0000", 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5039000000000001e-05 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134143+0000", 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7487999999999998e-05 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134292+0000", 2024-09-13T05:20:57.852 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000149641 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.134317+0000", 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4785000000000001e-05 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.136196+0000", 2024-09-13T05:20:57.853 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018790219999999999 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.658230950999993, 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.854 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.855 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.856 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:20:57.857 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.653987935000004, 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0041850189999999999, 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.858 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139456+0000", 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.859 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139454+0000", 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141751+0000", 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022969449999999999 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141785+0000", 2024-09-13T05:20:57.860 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4001000000000002e-05 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141813+0000", 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8183000000000001e-05 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141941+0000", 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012724500000000001 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.861 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.141962+0000", 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1041999999999999e-05 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143641+0000", 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016795969999999999 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:57.862 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.650111933000005, 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:20:57.864 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.865 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head, version: 46'5, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head@46'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:57.866 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.639219897999993, 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042548860000000003, 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154224+0000", 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.867 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.154203+0000", 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999785 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156290+0000", 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020874470000000001 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.868 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156324+0000", 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3212e-05 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156347+0000", 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3448999999999999e-05 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156533+0000", 2024-09-13T05:20:57.869 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018551599999999999 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.156570+0000", 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7858000000000002e-05 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.158479+0000", 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001908777 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 82.635527495000005, 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:20:57.872 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.873 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.874 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "pg_update_log_missing(2.5 epoch 104/98 rep_tid 120 entries 104'60 (0'0) error 2:a1fa0702:::rbd_object_map.11d2ef348a4.0000000000000008:head by client.6243.0:53 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 104'57)", 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:57.090489+0000", 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.70295483299999995, 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054663740000000004, 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:20:57.875 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:20:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:20:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090489+0000", 2024-09-13T05:20:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:20:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090489+0000", 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090489+0000", 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:20:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090499+0000", 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0457e-05 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090515+0000", 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6178999999999999e-05 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090525+0000", 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4080000000000004e-06 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090574+0000", 2024-09-13T05:20:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9675999999999997e-05 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.095956+0000", 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053814819999999999 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:20:57.879 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:20:57.881 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:57.878+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:57.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:20:57.981 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:57.976+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:58.017 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:58.017 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:58.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:20:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:20:58.088 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:58.084+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:58.182 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:58.180+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:58.198 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:58.198 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:58.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:20:58.302 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:58.300+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:58.384 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:58.378+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 161.24926672500001, 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065022250000000004, 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 5 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133323+0000", 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133319+0000", 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999962 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133337+0000", 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8447000000000001e-05 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133376+0000", 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8229000000000003e-05 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133393+0000", 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7713000000000001e-05 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133539+0000", 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014556200000000001 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.133558+0000", 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9057000000000001e-05 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.136871+0000", 2024-09-13T05:20:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003313084 2024-09-13T05:20:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137059+0000", 2024-09-13T05:20:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018830600000000001 2024-09-13T05:20:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.137312+0000", 2024-09-13T05:20:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025285800000000001 2024-09-13T05:20:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.138578+0000", 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012656799999999999 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139627+0000", 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001048605 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139716+0000", 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9038999999999999e-05 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:17.139825+0000", 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010922200000000001 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4190.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 160.23948519499999, 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064454229999999996, 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4190", 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/2754909258", 2024-09-13T05:20:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143104+0000", 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143101+0000", 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143124+0000", 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3343e-05 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143157+0000", 2024-09-13T05:20:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2706999999999997e-05 2024-09-13T05:20:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143175+0000", 2024-09-13T05:20:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8389e-05 2024-09-13T05:20:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143279+0000", 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010329 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.143293+0000", 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4005999999999999e-05 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146620+0000", 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033271239999999999 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146750+0000", 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130686 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.146981+0000", 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000230998 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.148216+0000", 2024-09-13T05:20:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001234684 2024-09-13T05:20:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149407+0000", 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011905399999999999 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149487+0000", 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0315999999999995e-05 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:18.149550+0000", 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2643000000000003e-05 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:44 2.7 e46/16 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head v 46'5, mlcod=46'3)", 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.016654942000002, 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085062139999999998, 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365934+0000", 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365932+0000", 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367890+0000", 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019577610000000001 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367927+0000", 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7243999999999998e-05 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.367944+0000", 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6892e-05 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368010+0000", 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5820000000000003e-05 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.368040+0000", 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9998999999999999e-05 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374382+0000", 2024-09-13T05:20:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063423330000000003 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374441+0000", 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8202000000000003e-05 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:48 2.6 2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.009625346999997, 2024-09-13T05:20:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012983517999999999, 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 48 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372964+0000", 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372959+0000", 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999952 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378255+0000", 2024-09-13T05:20:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052954919999999997 2024-09-13T05:20:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378285+0000", 2024-09-13T05:20:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0326000000000001e-05 2024-09-13T05:20:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378298+0000", 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2830000000000001e-05 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378376+0000", 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8312999999999994e-05 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014681199999999999 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385799+0000", 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072761960000000004 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385871+0000", 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1996999999999995e-05 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385947+0000", 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6415999999999995e-05 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:57 2.6 2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.968842511999995, 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019627206000000001, 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 57 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413747+0000", 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.413745+0000", 2024-09-13T05:20:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420545+0000", 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068001310000000001 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420578+0000", 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3256999999999997e-05 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420589+0000", 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0609e-05 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420659+0000", 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9858999999999996e-05 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420870+0000", 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021133799999999999 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433195+0000", 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012325153 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433274+0000", 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8487999999999998e-05 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.433374+0000", 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010034700000000001 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:58 2.6 2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.961984039000001, 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013563792999999999, 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 58 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420605+0000", 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.420603+0000", 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426835+0000", 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006231154 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426874+0000", 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9697999999999998e-05 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.426888+0000", 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4014e-05 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428405+0000", 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001516412 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428524+0000", 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011962 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434030+0000", 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005505149 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434081+0000", 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1875000000000001e-05 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.434169+0000", 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7651000000000005e-05 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4797.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 94.929842077999993, 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090560139999999994, 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4797", 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1084600622", 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452747+0000", 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452745+0000", 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452753+0000", 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452767+0000", 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4732999999999999e-05 2024-09-13T05:20:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452789+0000", 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1923999999999999e-05 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452853+0000", 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4219000000000003e-05 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.452861+0000", 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1949999999999997e-06 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454397+0000", 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001536471 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.454407+0000", 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9839999999999996e-06 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459379+0000", 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049716650000000001 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459514+0000", 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135551 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.459641+0000", 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126637 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461098+0000", 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014568840000000001 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461701+0000", 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00060324700000000001 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461760+0000", 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8882000000000003e-05 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:23.461803+0000", 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3257000000000003e-05 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:246 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2494464~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.315880604, 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040829689999999997, 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 246 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066709+0000", 2024-09-13T05:20:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066708+0000", 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066713+0000", 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066729+0000", 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5282e-05 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066738+0000", 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4909999999999994e-06 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066795+0000", 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6891999999999997e-05 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066803+0000", 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4139999999999997e-06 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070655+0000", 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038529580000000001 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070792+0000", 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013615099999999999 2024-09-13T05:20:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5052.0:247 2.2 2:54d0b850:::rbd_data.11d2ef348a4.0000000000000005:head [sparse-read 2625536~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.315719377999997, 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040411869999999999, 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5052", 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:20:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 247 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066870+0000", 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066869+0000", 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066874+0000", 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6119999999999998e-06 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066890+0000", 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5138e-05 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066897+0000", 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4039999999999998e-06 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066952+0000", 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5176000000000002e-05 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.066958+0000", 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7540000000000003e-06 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070814+0000", 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003855973 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:36.070911+0000", 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7162000000000002e-05 2024-09-13T05:20:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:80 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'83, mlcod=57'80)", 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.46698232, 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072315670000000004, 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915607+0000", 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915606+0000", 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915894+0000", 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028823899999999998 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915934+0000", 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9659000000000001e-05 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915953+0000", 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8660000000000001e-05 2024-09-13T05:20:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916091+0000", 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790800000000001 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916150+0000", 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9079000000000001e-05 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922698+0000", 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065481339999999997 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.922839+0000", 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140805 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:81 2.7 e65/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 65'84, mlcod=57'80)", 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.461771826000003, 2024-09-13T05:20:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003098127, 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920817+0000", 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920815+0000", 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920950+0000", 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502499999999999 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.920989+0000", 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8779000000000003e-05 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921013+0000", 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3753000000000001e-05 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921422+0000", 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040899799999999999 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.921481+0000", 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8811999999999999e-05 2024-09-13T05:20:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923815+0000", 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002333852 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923916+0000", 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101032 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:169 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'92, mlcod=65'89)", 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.052163241000002, 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004357662, 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330426+0000", 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330424+0000", 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330542+0000", 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117475 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330570+0000", 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8036999999999999e-05 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330585+0000", 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4983e-05 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330651+0000", 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6361999999999999e-05 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.330674+0000", 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3064000000000001e-05 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334660+0000", 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039862229999999997 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.334784+0000", 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012329499999999999 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:170 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'93, mlcod=65'89)", 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.046966592999993, 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035268080000000002, 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335623+0000", 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335621+0000", 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335688+0000", 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7006000000000002e-05 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335716+0000", 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.798e-05 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335728+0000", 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1712999999999999e-05 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335838+0000", 2024-09-13T05:20:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110575 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.335878+0000", 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9969000000000003e-05 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339094+0000", 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032155399999999998 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.339149+0000", 2024-09-13T05:20:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5685e-05 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:238 2.7 e66/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 66'101, mlcod=66'99)", 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 67.933495972000003, 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048419240000000001, 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449093+0000", 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.449091+0000", 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450611+0000", 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015204629999999999 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450655+0000", 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3662999999999999e-05 2024-09-13T05:20:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450676+0000", 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0573000000000001e-05 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450825+0000", 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014922399999999999 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.450859+0000", 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3779999999999998e-05 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453864+0000", 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030051959999999999 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:50.453935+0000", 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1302000000000003e-05 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5227.0:323 2.7 e70/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 70'103, mlcod=66'100)", 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.005345728999998, 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029606749999999999, 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377244+0000", 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377243+0000", 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377314+0000", 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1248999999999995e-05 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377333+0000", 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8924999999999999e-05 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377344+0000", 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0817e-05 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377395+0000", 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0803000000000001e-05 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.377416+0000", 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0993e-05 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380171+0000", 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027549660000000002 2024-09-13T05:20:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:58.380204+0000", 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3331999999999999e-05 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.5559.0:252 2.7 e73/16 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head v 73'147, mlcod=72'144)", 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.985267268000001, 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037496309999999998, 2024-09-13T05:20:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397322+0000", 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397329+0000", 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397341+0000", 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1321000000000001e-05 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397349+0000", 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2490000000000001e-06 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397396+0000", 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7172999999999997e-05 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.397420+0000", 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3828999999999998e-05 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401029+0000", 2024-09-13T05:20:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003609396 2024-09-13T05:20:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.401072+0000", 2024-09-13T05:20:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2308999999999998e-05 2024-09-13T05:20:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5559.0:258 2.2 2:58edda94:::rbd_data.11d2ef348a4.0000000000000001:head [write 3194880~999424] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.971038253000003, 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047026610000000003, 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5559", 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1174838758", 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 258 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411551+0000", 2024-09-13T05:20:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.411550+0000", 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412658+0000", 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001108314 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412672+0000", 2024-09-13T05:20:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3682999999999999e-05 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412678+0000", 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1739999999999997e-06 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412716+0000", 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7290999999999999e-05 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412768+0000", 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2710999999999997e-05 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.412844+0000", 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6157999999999998e-05 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.414915+0000", 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020706159999999999 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416189+0000", 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012737110000000001 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416226+0000", 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7082000000000002e-05 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:07.416254+0000", 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7840000000000001e-05 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5848.0:1 2.7 2:e6242823:::rbd_id.testrbdnbd21435:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:53.053207+0000", 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.3293820580000002, 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0030772459999999, 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5848", 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2166845676", 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:53.053207+0000", 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:53.053207+0000", 2024-09-13T05:20:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:53.053205+0000", 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:53.053217+0000", 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1517e-05 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:53.053238+0000", 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1115999999999999e-05 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:53.053254+0000", 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5974e-05 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:53.053383+0000", 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129231 2024-09-13T05:20:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:53.053407+0000", 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.419e-05 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.055735+0000", 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.002327588 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.056069+0000", 2024-09-13T05:20:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033428599999999999 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.056284+0000", 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000215527 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(osd.5.0:72 2.6 e104/93 2:695f0dc1:::rbd_data.11d2ef348a4.000000000000000a:7 v 104'53, mlcod=96'49)", 2024-09-13T05:20:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:57.071821+0000", 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.3107681369999999, 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032263919999999998, 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071821+0000", 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071821+0000", 2024-09-13T05:20:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071821+0000", 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071841+0000", 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0279999999999999e-05 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071850+0000", 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.054e-06 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071860+0000", 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071928+0000", 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.813e-05 2024-09-13T05:20:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071952+0000", 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4406999999999999e-05 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.075002+0000", 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030494659999999998 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.075048+0000", 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5711999999999999e-05 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6243.0:53 2.5 2:a1fa0702:::rbd_object_map.11d2ef348a4.0000000000000008:head [call rbd.object_map_snap_remove in=25b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e104)", 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:57.089496+0000", 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.293093184, 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061061290000000001, 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:20:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6243", 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1368778665", 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 53 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.089496+0000", 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.089496+0000", 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.089495+0000", 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.089503+0000", 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9559999999999997e-06 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.089513+0000", 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7359999999999996e-06 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.089520+0000", 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7939999999999997e-06 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.089549+0000", 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8651000000000001e-05 2024-09-13T05:20:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.089600+0000", 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1693000000000001e-05 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.095602+0000", 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060019729999999999 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:58.493 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:58.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:20:58.495 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:58.490+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:58.595 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:58.592+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:58.668 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:58.668 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:58.668 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:58.668 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:58.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:20:58.696 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:58.694+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:58.796 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:58.794+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:58.853 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:58.853 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:58.853 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:20:58.853 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:20:58.853 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:58.868 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:20:58.918 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:20:58.916+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:20:58.998 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:58.998 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 168.47899529099999, 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:20:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:20:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 168.478723266, 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 168.47867968, 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:20:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 168.47864745300001, 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 168.478436305, 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 168.47840001200001, 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 168.478361562, 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:20:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:20:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 168.47818666000001, 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.663482721999998, 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.655952608999996, 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.646383282000002, 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:20:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:20:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.635220410000002, 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.629060812999995, 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:20:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:20:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:20:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:20:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.068 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.580225033000005, 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.571948499000001, 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.074 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:20:59.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:20:59.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:20:59.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.079188091000006, 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.079 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:59.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:20:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:20:59.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.078752944000001, 2024-09-13T05:20:59.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.089 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.078438951999999, 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:20:59.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.078033700999995, 2024-09-13T05:20:59.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:20:59.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:20:59.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.077709162999994, 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:20:59.101 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:20:59.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:20:59.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:20:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:20:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:20:59.105 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:20:59.105 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:59.106 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:58.994+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:59.106 INFO:tasks.workunit.client.0.smithi122.stderr: Removing all snapshots: 25% complete... Removing all snapshots: 50% complete... Removing all snapshots: 75% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T05:20:59.107 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:59.102+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:59.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:20:59.110 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:20:59.108+0000 7f2c2fd03640 0 -- 172.21.15.122:0/1368778665 >> [v2:172.21.15.16:6816/1006404377,v1:172.21.15.16:6817/1006404377] conn(0x7f2c0c008bf0 msgr2=0x7f2c0c00b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T05:20:59.119 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace '' remove testrbdnbd21435 2024-09-13T05:20:59.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T05:20:59.207 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:59.202+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:59.285 INFO:tasks.workunit.client.0.smithi122.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-13T05:20:59.280+0000 7f175e0b9640 0 -- 172.21.15.122:0/1422905144 >> [v2:172.21.15.16:6800/2153409514,v1:172.21.15.16:6801/2153409514] conn(0x564dc4165470 msgr2=0x564dc4167900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T05:20:59.285 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:20:59.286 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:20:59.286 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:20:59.286 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:20:59.304 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:20:59.307 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:20:59.302+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:20:59.308 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:20:59.309 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:20:59.312 INFO:tasks.workunit.client.0.smithi122.stderr: Removing image: 100% complete...done. 2024-09-13T05:20:59.321 INFO:tasks.workunit.client.0.smithi122.stderr:+ for ns in '' ${NS} 2024-09-13T05:20:59.321 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace ns status testrbdnbd21435 2024-09-13T05:20:59.326 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:20:59.326 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:20:59.342 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:20:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:20:59.362 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:20:59.363 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:20:59.382 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:20:59.382 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:20:59.386 INFO:tasks.workunit.client.0.smithi122.stdout:Watchers: none 2024-09-13T05:20:59.390 INFO:tasks.workunit.client.0.smithi122.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-13T05:20:59.390 INFO:tasks.workunit.client.0.smithi122.stderr:+ sleep 0.5 2024-09-13T05:20:59.402 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:20:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:20:59.408 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:20:59.404+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:20:59.421 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:59.421 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:59.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:20:59.509 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:20:59.506+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:20:59.593 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:59.593 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:59.608 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:20:59.609 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:59.604+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:59.710 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:59.706+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:59.739 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:59.739 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:59.753 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:20:59.844 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:20:59.842+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:20:59.892 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace ns status testrbdnbd21435 2024-09-13T05:20:59.892 INFO:tasks.workunit.client.0.smithi122.stderr:+ grep 'Watchers: none' 2024-09-13T05:20:59.912 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:20:59.908+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:20:59.917 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:20:59.917 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:20:59.932 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:20:59.955 INFO:tasks.workunit.client.0.smithi122.stdout:Watchers: none 2024-09-13T05:20:59.956 INFO:tasks.workunit.client.0.smithi122.stderr:+ break 2024-09-13T05:20:59.956 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace ns snap purge testrbdnbd21435 2024-09-13T05:21:00.013 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:21:00.008+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:21:00.098 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:00.098 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:00.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T05:21:00.113 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:21:00.110+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:21:00.116 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:21:00.119 INFO:tasks.workunit.client.0.smithi122.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T05:21:00.134 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd -p rbd --namespace ns remove testrbdnbd21435 2024-09-13T05:21:00.215 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:21:00.212+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:21:00.266 INFO:tasks.workunit.client.0.smithi122.stderr:2024-09-13T05:21:00.264+0000 7f6e10681640 0 -- 172.21.15.122:0/3408095837 >> [v2:172.21.15.79:6800/2778785304,v1:172.21.15.79:6801/2778785304] conn(0x56200053a8b0 msgr2=0x56200053cca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T05:21:00.285 INFO:tasks.workunit.client.0.smithi122.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-13T05:21:00.280+0000 7f6e0f41f640 0 -- 172.21.15.122:0/3408095837 >> [v2:172.21.15.16:6800/2153409514,v1:172.21.15.16:6801/2153409514] conn(0x56200058bd80 msgr2=0x56200058e210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T05:21:00.302 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:00.303 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:00.306 INFO:tasks.workunit.client.0.smithi122.stderr: Removing image: 100% complete...done. 2024-09-13T05:21:00.315 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd namespace remove rbd/ns 2024-09-13T05:21:00.316 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:21:00.312+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:21:00.318 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:21:00.378 INFO:tasks.workunit.client.0.smithi122.stderr:+ rbd config global rm global rbd_default_pool 2024-09-13T05:21:00.414 INFO:tasks.workunit.client.0.smithi122.stderr:+ ceph osd pool delete new_default_pool21435 new_default_pool21435 --yes-i-really-really-mean-it 2024-09-13T05:21:00.416 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:21:00.412+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:21:00.436 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:00.436 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:00.450 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:21:00.517 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:21:00.512+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:21:00.573 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:00.573 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:00.587 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:21:00.636 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:21:00.634+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:21:00.704 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:00.705 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:00.718 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:21:00.719 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:21:00.714+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:21:00.819 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:21:00.814+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:21:00.875 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:00.876 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:00.890 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:21:00.920 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:21:00.916+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:21:01.020 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-13T05:21:01.016+0000 7feb42bf7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25006) UID: 0 2024-09-13T05:21:01.049 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:01.049 INFO:teuthology.orchestra.run.smithi079.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:01.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T05:21:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-13T05:21:01.121 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:21:01.118+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:21:01.195 INFO:tasks.workunit.client.0.smithi122.stderr:pool 'new_default_pool21435' does not exist 2024-09-13T05:21:01.203 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:01.203 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:01.211 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T05:21:01.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T05:21:01.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T05:21:01.222 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-09-13T05:21:01.218+0000 7f7db1f39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32423) UID: 0 2024-09-13T05:21:01.265 INFO:tasks.workunit:Stopping ['rbd/rbd-nbd.sh'] on client.0... 2024-09-13T05:21:01.265 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T05:21:01.323 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-09-13T05:21:01.318+0000 7f5354565640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 36813) UID: 0 2024-09-13T05:21:01.343 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:01.343 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:01.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T05:21:01.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T05:21:01.423 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:21:01.420+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:21:01.480 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-13T05:21:01.480 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T05:21:01.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T05:21:01.538 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-13T05:21:01.536+0000 7fb3b0f2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25004) UID: 0 2024-09-13T05:21:01.566 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-13T05:21:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T05:21:01.583 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-13T05:21:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T05:21:01.599 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-13T05:21:01.600 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T05:21:01.612 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:21:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:21:01.612 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:01.616 DEBUG:teuthology.parallel:result is None 2024-09-13T05:21:01.616 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T05:21:01.620 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-13T05:21:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T05:21:01.625 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-09-13T05:21:01.622+0000 7f4a837c7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25194) UID: 0 2024-09-13T05:21:01.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T05:21:01.628 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T05:21:01.628 DEBUG:teuthology.orchestra.run.smithi122:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T05:21:01.641 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-13T05:21:01.641 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T05:21:01.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-13T05:21:01.663 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T05:21:01.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-13T05:21:01.724+0000 7f133e485640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25002) UID: 0 2024-09-13T05:21:01.758 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:21:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:21:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:21:01.758 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:01.774 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T05:21:01.774 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-13T05:21:01.783 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T05:21:01.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T05:21:01.942 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:21:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:21:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:21:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:21:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:21:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 171.42295105400001, 2024-09-13T05:21:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014247040000000001, 2024-09-13T05:21:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:21:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:21:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 134 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515444+0000", 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515442+0000", 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515650+0000", 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020713200000000001 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515679+0000", 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9473000000000001e-05 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515697+0000", 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7537000000000002e-05 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.521870+0000", 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061734379999999998 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522044+0000", 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173685 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.522257+0000", 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021336900000000001 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.527796+0000", 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055389209999999996 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529563+0000", 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001767123 2024-09-13T05:21:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529636+0000", 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2602999999999996e-05 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529691+0000", 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5402000000000003e-05 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 171.42267902899999, 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014272959999999999, 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 135 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:21:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515716+0000", 2024-09-13T05:21:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515715+0000", 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515724+0000", 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1730000000000001e-06 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515740+0000", 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5549999999999999e-05 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515752+0000", 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.167e-05 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523750+0000", 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079989280000000006 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.523894+0000", 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014336199999999999 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.524096+0000", 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020259600000000001 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528318+0000", 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042220599999999997 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529824+0000", 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001505869 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529894+0000", 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9856000000000003e-05 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.529989+0000", 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4865999999999996e-05 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 171.42263544299999, 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014573641, 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 136 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515760+0000", 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515759+0000", 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515766+0000", 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4640000000000003e-06 2024-09-13T05:21:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515775+0000", 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2329999999999995e-06 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515782+0000", 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0770000000000002e-06 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525117+0000", 2024-09-13T05:21:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093350740000000005 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525349+0000", 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023176499999999999 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.525555+0000", 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000205832 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528339+0000", 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027846049999999999 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530131+0000", 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001792383 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530213+0000", 2024-09-13T05:21:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1151999999999998e-05 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530333+0000", 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000120716 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:21:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 171.422603216, 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014916798, 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 137 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515792+0000", 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515791+0000", 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515966+0000", 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017471500000000001 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515982+0000", 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6699999999999999e-05 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.515992+0000", 2024-09-13T05:21:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1460000000000002e-06 2024-09-13T05:21:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526286+0000", 2024-09-13T05:21:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010294005 2024-09-13T05:21:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526417+0000", 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131509 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.526628+0000", 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021045100000000001 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530354+0000", 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037263880000000002 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530484+0000", 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130479 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530611+0000", 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012635500000000001 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530709+0000", 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7940000000000001e-05 2024-09-13T05:21:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 171.42239206799999, 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018401191000000001, 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 138 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516003+0000", 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516002+0000", 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516012+0000", 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5230000000000001e-06 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516021+0000", 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5470000000000002e-06 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516028+0000", 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9099999999999999e-06 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528457+0000", 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012429300000000001 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528638+0000", 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000181039 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.528837+0000", 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019867800000000001 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531691+0000", 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028537319999999999 2024-09-13T05:21:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534188+0000", 2024-09-13T05:21:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024969039999999999 2024-09-13T05:21:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534300+0000", 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112372 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534404+0000", 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104109 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:21:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 171.422355775, 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019616801999999999, 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 139 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:21:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516039+0000", 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516050+0000", 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516060+0000", 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6299999999999993e-06 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516069+0000", 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.696e-06 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530773+0000", 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014703486999999999 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.530894+0000", 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012109500000000001 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531097+0000", 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020303199999999999 2024-09-13T05:21:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534551+0000", 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034534069999999999 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535490+0000", 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093937600000000001 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535555+0000", 2024-09-13T05:21:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5112999999999998e-05 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535656+0000", 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101189 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:21:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 171.42231732499999, 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019972495, 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 140 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516078+0000", 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516077+0000", 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516222+0000", 2024-09-13T05:21:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014457200000000001 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516233+0000", 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1486e-05 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516241+0000", 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7440000000000007e-06 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.531794+0000", 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015552623 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532041+0000", 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024755799999999999 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.532464+0000", 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042302399999999998 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535685+0000", 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032203079999999999 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535786+0000", 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101279 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.535949+0000", 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163449 2024-09-13T05:21:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536050+0000", 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010105199999999999 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 171.422142423, 2024-09-13T05:21:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020148827000000001, 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4195", 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.79:0/1032918293", 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 141 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516253+0000", 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516252+0000", 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516262+0000", 2024-09-13T05:21:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5130000000000002e-06 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516271+0000", 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6169999999999996e-06 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.516278+0000", 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2130000000000002e-06 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534598+0000", 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018320100999999998 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534725+0000", 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126446 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.534946+0000", 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022082199999999999 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536149+0000", 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001203024 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536210+0000", 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000001e-05 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536299+0000", 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8615000000000005e-05 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:18:10.536402+0000", 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102716 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:36 2.3 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.607438485, 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017162572000000001, 2024-09-13T05:21:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 36 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330957+0000", 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.330955+0000", 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338434+0000", 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007478625 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338457+0000", 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553999999999998e-05 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338467+0000", 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0091999999999999e-05 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338528+0000", 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1066000000000001e-05 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338652+0000", 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012356800000000001 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338750+0000", 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7969999999999999e-05 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.345223+0000", 2024-09-13T05:21:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064734140000000003 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348029+0000", 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002806102 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348078+0000", 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8675999999999999e-05 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348119+0000", 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1202999999999998e-05 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:37 2.5 2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.599908372, 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016738367000000001, 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 37 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338487+0000", 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338485+0000", 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348015+0000", 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095296419999999996 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348030+0000", 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348039+0000", 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.268e-06 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348105+0000", 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5630999999999996e-05 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348242+0000", 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137145 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355050+0000", 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068073860000000003 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355124+0000", 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4029999999999994e-05 2024-09-13T05:21:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355225+0000", 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010148 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:39 2.5 2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.59033904499999, 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013964518, 2024-09-13T05:21:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 39 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348056+0000", 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.348054+0000", 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355269+0000", 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0072142689999999997 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355283+0000", 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3959e-05 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355292+0000", 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5140000000000007e-06 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355335+0000", 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3346999999999997e-05 2024-09-13T05:21:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355503+0000", 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000167818 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361864+0000", 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063609280000000001 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.361932+0000", 2024-09-13T05:21:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7263999999999998e-05 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.362021+0000", 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9123999999999996e-05 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:44 2.7 2:e0ae2972:ns::rbd_data.11db3aaeebba.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.57917617299999, 2024-09-13T05:21:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019491294999999999, 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 44 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359219+0000", 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.359217+0000", 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365313+0000", 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060962769999999998 2024-09-13T05:21:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365344+0000", 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0392999999999999e-05 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365357+0000", 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2819e-05 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365428+0000", 2024-09-13T05:21:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1779999999999994e-05 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365622+0000", 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019415 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365798+0000", 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017554699999999999 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375370+0000", 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095715980000000006 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378564+0000", 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031947640000000001 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378660+0000", 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6086000000000001e-05 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378710+0000", 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9799999999999998e-05 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:45 2.3 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.573016576, 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020858953999999999, 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 45 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365379+0000", 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.365377+0000", 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371680+0000", 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0063030580000000003 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371706+0000", 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5896000000000001e-05 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371722+0000", 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951999999999999e-05 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371799+0000", 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7577999999999995e-05 2024-09-13T05:21:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371978+0000", 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017859000000000001 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 2", 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372155+0000", 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177176 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385665+0000", 2024-09-13T05:21:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013509989 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386116+0000", 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045136999999999998 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386176+0000", 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0066999999999999e-05 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.386238+0000", 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1177000000000001e-05 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:59 2.5 2:b65d9fef:ns::rbd_data.11db3aaeebba.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.524180796, 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021670004, 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 59 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414214+0000", 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.414212+0000", 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422399+0000", 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081863490000000008 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422445+0000", 2024-09-13T05:21:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6418999999999997e-05 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422464+0000", 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8926000000000001e-05 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422589+0000", 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012487500000000001 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422892+0000", 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030345300000000002 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435660+0000", 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012767390999999999 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435750+0000", 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0523e-05 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.435884+0000", 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134184 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:60 2.5 2:b8e782ee:ns::rbd_data.11db3aaeebba.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.51590426200001, 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014915637000000001, 2024-09-13T05:21:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 60 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422491+0000", 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.422489+0000", 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428682+0000", 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006193385 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428727+0000", 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4832e-05 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428755+0000", 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8529000000000001e-05 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.428847+0000", 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2070000000000004e-05 2024-09-13T05:21:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.429087+0000", 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023937200000000001 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437199+0000", 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081117689999999996 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437279+0000", 2024-09-13T05:21:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0270000000000002e-05 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.437407+0000", 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012766700000000001 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:83 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.023143853999997, 2024-09-13T05:21:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014213851, 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 83 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915251+0000", 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915250+0000", 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915594+0000", 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034413100000000002 2024-09-13T05:21:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915640+0000", 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6526999999999997e-05 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915672+0000", 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1736000000000002e-05 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923427+0000", 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077545260000000003 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923668+0000", 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000241733 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.923953+0000", 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028435199999999998 2024-09-13T05:21:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927300+0000", 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033475979999999998 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929237+0000", 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001936501 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929369+0000", 2024-09-13T05:21:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131991 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929465+0000", 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6286000000000006e-05 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:84 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.022708707000007, 2024-09-13T05:21:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017034239, 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 84 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915686+0000", 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915685+0000", 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915950+0000", 2024-09-13T05:21:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026477899999999998 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915969+0000", 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.912e-05 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915985+0000", 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5849999999999999e-05 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926043+0000", 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010057961000000001 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926156+0000", 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113105 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.926374+0000", 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021781 2024-09-13T05:21:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929622+0000", 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032475960000000002 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932448+0000", 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028262949999999999 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932595+0000", 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014748800000000001 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.932721+0000", 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012538800000000001 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:85 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:21:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.022394715000004, 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018564082999999999, 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 85 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:21:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916000+0000", 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.915999+0000", 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916300+0000", 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030135300000000003 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916337+0000", 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6596e-05 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916360+0000", 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.323e-05 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927403+0000", 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011043083 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.927650+0000", 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024631099999999998 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.928025+0000", 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000375464 2024-09-13T05:21:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931756+0000", 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037308319999999999 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934423+0000", 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002666767 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934486+0000", 2024-09-13T05:21:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3305999999999993e-05 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934565+0000", 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8535000000000006e-05 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:86 2.7 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:21:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.021989464000001, 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018444327999999999, 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 86 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916406+0000", 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916404+0000", 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916669+0000", 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026543600000000002 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916696+0000", 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6727e-05 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916713+0000", 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6526000000000001e-05 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929716+0000", 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013003768000000001 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.929924+0000", 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207147 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.930315+0000", 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039137200000000002 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934593+0000", 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042776100000000003 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934672+0000", 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9676000000000001e-05 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934733+0000", 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0299e-05 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.934850+0000", 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117501 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5227.0:87 2.7 2.f93e1baf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.021664926, 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014793318999999999, 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5227", 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2840937132", 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 87 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916730+0000", 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916729+0000", 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.916981+0000", 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025222299999999999 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917012+0000", 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1130000000000002e-05 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.917030+0000", 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8272999999999999e-05 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931497+0000", 2024-09-13T05:21:02.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014466443000000001 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:49.931524+0000", 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6760000000000001e-05 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.058 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:02.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T05:21:02.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T05:21:02.193 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:21:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-13T05:21:02.193 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:02.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T05:21:02.279 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a: {} 2024-09-13T05:21:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T05:21:02.292 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b: {} 2024-09-13T05:21:02.293 INFO:teuthology.orchestra.run.smithi016.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T05:21:02.303 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c: {} 2024-09-13T05:21:02.303 INFO:teuthology.orchestra.run.smithi016.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T05:21:02.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T05:21:02.328 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:02.328 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-13T05:21:02.328 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-13T05:21:02.328 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-13T05:21:02.328 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:02.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T05:21:02.506 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:02.507 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-13T05:21:02.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-13T05:21:02.507 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-13T05:21:02.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.507 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:02.507 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:21:02.507 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.177135963, 2024-09-13T05:21:02.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028364350999999999, 2024-09-13T05:21:02.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327656+0000", 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.327654+0000", 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332423+0000", 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047686569999999999 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332455+0000", 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.146e-05 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332469+0000", 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.468e-05 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332538+0000", 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9076000000000001e-05 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332665+0000", 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126611 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.332767+0000", 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102394 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.337550+0000", 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047823400000000004 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355907+0000", 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018357616 2024-09-13T05:21:02.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.355970+0000", 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2941000000000006e-05 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356020+0000", 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0009000000000002e-05 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.513 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:36 2.3 e46/16 2:c7165869:ns::rbd_data.11db3aaeebba.0000000000000001:head v 46'28, mlcod=46'26)", 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.165930202, 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089648599999999998, 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:21:02.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338862+0000", 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.338861+0000", 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341230+0000", 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023693379999999999 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341253+0000", 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3366e-05 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341264+0000", 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0318e-05 2024-09-13T05:21:02.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341342+0000", 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8745000000000006e-05 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.341363+0000", 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0675000000000001e-05 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347767+0000", 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064038699999999999 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.347827+0000", 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9509000000000002e-05 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:35 2.0 2:068e802e:ns::rbd_data.11db3aaeebba.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.15575114799999, 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075867649999999997, 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 35 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349041+0000", 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.349039+0000", 2024-09-13T05:21:02.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356432+0000", 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073930569999999998 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356452+0000", 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9453000000000002e-05 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356462+0000", 2024-09-13T05:21:02.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0752e-05 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356525+0000", 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2723000000000002e-05 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356628+0000", 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000102342 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.522 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:38 2.0 2:0ac68db7:ns::rbd_data.11db3aaeebba.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.148315345, 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028066282000000001, 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:02.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 38 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356477+0000", 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.356475+0000", 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360018+0000", 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003543095 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360048+0000", 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0063e-05 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360063+0000", 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4793999999999999e-05 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360126+0000", 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2677999999999999e-05 2024-09-13T05:21:02.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360325+0000", 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019928599999999999 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.360507+0000", 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018213300000000001 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370690+0000", 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010182535 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384474+0000", 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013784453 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384515+0000", 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0457999999999998e-05 2024-09-13T05:21:02.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.384543+0000", 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8214000000000001e-05 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4571.0:46 2.0 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:21:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.141707642, 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024907219000000001, 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4571", 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 46 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363084+0000", 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.363083+0000", 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370584+0000", 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075014590000000003 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370620+0000", 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6010000000000003e-05 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370639+0000", 2024-09-13T05:21:02.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8512000000000002e-05 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370755+0000", 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116209 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.370936+0000", 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018084199999999999 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.371106+0000", 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017015199999999999 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.378191+0000", 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070848309999999998 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387926+0000", 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097351590000000002 2024-09-13T05:21:02.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387964+0000", 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8544999999999998e-05 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.387991+0000", 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7019999999999999e-05 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4571.0:45 2.3 e46/16 2:dd295971:ns::rbd_data.11db3aaeebba.0000000000000007:head v 46'29, mlcod=46'27)", 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.13248142400001, 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013649705, 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372310+0000", 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.372309+0000", 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374865+0000", 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025557119999999999 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374882+0000", 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7085999999999999e-05 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374891+0000", 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9779999999999994e-06 2024-09-13T05:21:02.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.374982+0000", 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1283e-05 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.375012+0000", 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0298e-05 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385907+0000", 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010894355999999999 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:19:21.385960+0000", 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3597000000000002e-05 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.540 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head, version: 46'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head@46'19, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.517407928000001, 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003044452, 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987384+0000", 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.987382+0000", 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988524+0000", 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011418859999999999 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988545+0000", 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0483e-05 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988556+0000", 2024-09-13T05:21:02.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1586999999999999e-05 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988662+0000", 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106137 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.988680+0000", 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7295999999999998e-05 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:30.990428+0000", 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017486789999999999 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head, version: 46'20, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head@46'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.483643085000001, 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030823669999999999, 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021149+0000", 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.021147+0000", 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022523+0000", 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013751779999999999 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022563+0000", 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0009999999999998e-05 2024-09-13T05:21:02.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022591+0000", 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8683e-05 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022701+0000", 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110173 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.022721+0000", 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9911e-05 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.024231+0000", 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001509886 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.549 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.474588331, 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031121460000000001, 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030204+0000", 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.030202+0000", 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031555+0000", 2024-09-13T05:21:02.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001352518 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031595+0000", 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0197000000000002e-05 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031646+0000", 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0742000000000003e-05 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031787+0000", 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014145999999999999 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.031806+0000", 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8845e-05 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.033316+0000", 2024-09-13T05:21:02.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015096739999999999 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7, version: 72'63, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7@72'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.457070229999999, 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038240729999999999, 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047722+0000", 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.047720+0000", 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049066+0000", 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013460589999999999 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049110+0000", 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3702999999999998e-05 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049137+0000", 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7443000000000001e-05 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049277+0000", 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013948399999999999 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.049298+0000", 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1234e-05 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.051546+0000", 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022478400000000001 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.558 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.440758948999999, 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030803480000000001, 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064033+0000", 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.064032+0000", 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065378+0000", 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013465630000000001 2024-09-13T05:21:02.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065418+0000", 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0422e-05 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065446+0000", 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7477000000000001e-05 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065577+0000", 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131254 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.065599+0000", 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1656999999999999e-05 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.067113+0000", 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001514409 2024-09-13T05:21:02.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.434496831000001, 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030880719999999999, 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070295+0000", 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.070294+0000", 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071663+0000", 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001369089 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071704+0000", 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0745999999999999e-05 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071731+0000", 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7182000000000002e-05 2024-09-13T05:21:02.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071865+0000", 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013386800000000001 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.071885+0000", 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0800000000000001e-05 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.073383+0000", 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001497757 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPush(2.4 88/87 [PushOp(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7, version: 72'65, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7@72'65, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.428148617000002, 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033570940000000001, 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076643+0000", 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.076642+0000", 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077619+0000", 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00097751200000000004 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077660+0000", 2024-09-13T05:21:02.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0368999999999999e-05 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077690+0000", 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0139999999999999e-05 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077823+0000", 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013348400000000001 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.077842+0000", 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8255999999999999e-05 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:31.080000+0000", 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021586169999999998 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.571 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5848.0:12 2.3 2:d073633c:::rbd_header.11d2ef348a4:head [call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b,call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b,call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b,call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:54.075088+0000", 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.4297043489999997, 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035789509999999999, 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5848", 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/2166845676", 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.075088+0000", 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.075088+0000", 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.075085+0000", 2024-09-13T05:21:02.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.075103+0000", 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8669e-05 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.075140+0000", 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6789999999999998e-05 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.075162+0000", 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1829999999999999e-05 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.075283+0000", 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121191 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.075621+0000", 2024-09-13T05:21:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033810600000000002 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:54.078666+0000", 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030451150000000001 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6243.0:41 2.3 2:d073633c:::rbd_header.11d2ef348a4:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e104)", 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:57.070855+0000", 2024-09-13T05:21:02.576 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4339364720000001, 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044446809999999998, 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6243", 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1368778665", 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 41 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.070855+0000", 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.070855+0000", 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.070853+0000", 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.070864+0000", 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0845e-05 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.070877+0000", 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.306e-05 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.070887+0000", 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5859999999999994e-06 2024-09-13T05:21:02.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071333+0000", 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044580500000000002 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071465+0000", 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132652 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:21:02.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.071885+0000", 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00041960199999999998 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.072789+0000", 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00090394 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.075208+0000", 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024186089999999999 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.075261+0000", 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3093999999999997e-05 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.075300+0000", 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9440999999999997e-05 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6243.0:50 2.3 2:d073633c:::rbd_header.11d2ef348a4:head [call rbd.snapshot_trash_add in=21b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e104)", 2024-09-13T05:21:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:57.082205+0000", 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.4225866030000001, 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057984229999999996, 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6243", 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1368778665", 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 50 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.082205+0000", 2024-09-13T05:21:02.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.082205+0000", 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.082204+0000", 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.082211+0000", 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6710000000000002e-06 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.082227+0000", 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6064000000000001e-05 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.082243+0000", 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6197000000000001e-05 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.082295+0000", 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2324000000000003e-05 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.082389+0000", 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3882999999999995e-05 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.082704+0000", 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00031481199999999998 2024-09-13T05:21:02.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.083460+0000", 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075623599999999997 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.087907+0000", 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044463009999999997 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.087961+0000", 2024-09-13T05:21:02.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3878000000000002e-05 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:57.088004+0000", 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3142999999999997e-05 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6243.0:56 2.3 2:d073633c:::rbd_header.11d2ef348a4:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e105)", 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:58.081911+0000", 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.4228808820000003, 2024-09-13T05:21:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014155776, 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6243", 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1368778665", 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 56 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.081911+0000", 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.081911+0000", 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.081909+0000", 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.081922+0000", 2024-09-13T05:21:02.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3396e-05 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.081947+0000", 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5258000000000001e-05 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.081960+0000", 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2452e-05 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.082098+0000", 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000137923 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.082117+0000", 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9656999999999998e-05 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.086655+0000", 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004537157 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.086762+0000", 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107453 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.087076+0000", 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00031400900000000002 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.091138+0000", 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040615790000000001 2024-09-13T05:21:02.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.095972+0000", 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048344360000000001 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.096030+0000", 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7917999999999997e-05 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.096067+0000", 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6752000000000003e-05 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6243.0:65 2.3 2:d073633c:::rbd_header.11d2ef348a4:head [call rbd.snapshot_trash_add in=21b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e105)", 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:58.103660+0000", 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.401132091, 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056856570000000002, 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6243", 2024-09-13T05:21:02.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1368778665", 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 65 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.103660+0000", 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.103660+0000", 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.103659+0000", 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.103664+0000", 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8690000000000003e-06 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.103677+0000", 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2601e-05 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.103685+0000", 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1179999999999992e-06 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.103735+0000", 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9901999999999998e-05 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.103823+0000", 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7967000000000007e-05 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.104158+0000", 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033582799999999998 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.105079+0000", 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00092054599999999995 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.109250+0000", 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041712019999999997 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.109304+0000", 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4048999999999997e-05 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:58.109345+0000", 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1319000000000002e-05 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6243.0:71 2.3 2:d073633c:::rbd_header.11d2ef348a4:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e106)", 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:59.094905+0000", 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.4098869660000002, 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003286012, 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6243", 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1368778665", 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 71 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.601 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.094905+0000", 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.094905+0000", 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.094902+0000", 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2024-09-13T05:21:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.094921+0000", 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8553e-05 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.094954+0000", 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2756000000000003e-05 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.094976+0000", 2024-09-13T05:21:02.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.245e-05 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.095204+0000", 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000227994 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.095352+0000", 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147985 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.095915+0000", 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00056343699999999997 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.097377+0000", 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014613880000000001 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.098033+0000", 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000656289 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.098122+0000", 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8932999999999997e-05 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.098191+0000", 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8953000000000002e-05 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.606 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6243.0:77 2.3 2:d073633c:::rbd_header.11d2ef348a4:head [watch unwatch cookie 139827223814640] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e106)", 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-13T05:20:59.107032+0000", 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.397759803, 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003105357, 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6243", 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.122:0/1368778665", 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 77 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-13T05:21:02.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.107032+0000", 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.107032+0000", 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.107031+0000", 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.107038+0000", 2024-09-13T05:21:02.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6030000000000001e-06 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.107051+0000", 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3193999999999999e-05 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.107059+0000", 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6729999999999992e-06 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-13T05:21:02.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.107556+0000", 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000496818 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.107681+0000", 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000125005 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.107904+0000", 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223828 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2024-09-13T05:21:02.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.109165+0000", 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001260513 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.109893+0000", 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00072829199999999996 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.109985+0000", 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1939000000000002e-05 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-13T05:20:59.110137+0000", 2024-09-13T05:21:02.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000152299 2024-09-13T05:21:02.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.612 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-13T05:21:02.612 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-13T05:21:02.612 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:02.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T05:21:02.744 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:21:02.745 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:21:02.745 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:21:02.745 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:21:02.758 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T05:21:02.881 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:21:02.882 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:21:02.882 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:21:02.882 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:21:02.882 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:21:02.896 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:105 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'67, mlcod=14'64)", 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 172.887553645, 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.006714238, 2024-09-13T05:21:03.023 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132526+0000", 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132524+0000", 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132539+0000", 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4012999999999999e-05 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132556+0000", 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7045999999999999e-05 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132565+0000", 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6800000000000005e-06 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132927+0000", 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000361431 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.132970+0000", 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3439000000000002e-05 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139176+0000", 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062060420000000002 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139240+0000", 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.4022999999999993e-05 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:106 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'68, mlcod=14'64)", 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 172.88692238600001, 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084010779999999993, 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133157+0000", 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133155+0000", 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133256+0000", 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010077900000000001 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133305+0000", 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.8353000000000003e-05 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133325+0000", 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0893000000000001e-05 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133679+0000", 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00035306399999999999 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133759+0000", 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.0161999999999996e-05 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141485+0000", 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0077266369999999997 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141558+0000", 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2910000000000005e-05 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:107 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'69, mlcod=14'64)", 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 172.88627726499999, 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0078160829999999997, 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.032 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133802+0000", 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133801+0000", 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133812+0000", 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.077e-05 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133829+0000", 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6803000000000002e-05 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.034 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.133839+0000", 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0647e-05 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134748+0000", 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00090876300000000004 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134802+0000", 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4166000000000003e-05 2024-09-13T05:21:03.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141566+0000", 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0067637069999999999 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141618+0000", 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.2370000000000002e-05 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:108 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'70, mlcod=14'64)", 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 172.88576990199999, 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0073820489999999999, 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134310+0000", 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134308+0000", 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134317+0000", 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.8270000000000004e-06 2024-09-13T05:21:03.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134335+0000", 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7200000000000001e-05 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134348+0000", 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3115999999999999e-05 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135378+0000", 2024-09-13T05:21:03.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010306790000000001 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135411+0000", 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3015999999999997e-05 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141629+0000", 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062173719999999997 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141692+0000", 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3015999999999994e-05 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:109 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'71, mlcod=14'64)", 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 172.88524446599999, 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069214630000000001, 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134835+0000", 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134834+0000", 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134894+0000", 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9825999999999999e-05 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134909+0000", 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5353e-05 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.134920+0000", 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0682999999999999e-05 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136002+0000", 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001082265 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136052+0000", 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.0003000000000002e-05 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141701+0000", 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005648793 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141757+0000", 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5633000000000001e-05 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:110 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'72, mlcod=14'64)", 2024-09-13T05:21:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 172.88460695399999, 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0063475839999999999, 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135473+0000", 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135471+0000", 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135483+0000", 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1677e-05 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135508+0000", 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5063e-05 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.135522+0000", 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3896e-05 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137055+0000", 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001532678 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6493000000000003e-05 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141763+0000", 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0046516109999999999 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141820+0000", 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7373000000000001e-05 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.049 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:111 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'73, mlcod=14'65)", 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 172.88403024600001, 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058411579999999999, 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136049+0000", 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136048+0000", 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136058+0000", 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.876e-06 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136088+0000", 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0154e-05 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.136099+0000", 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0482999999999999e-05 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137719+0000", 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001620661 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137755+0000", 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5769000000000003e-05 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141829+0000", 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004073442 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.141890+0000", 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1855999999999999e-05 2024-09-13T05:21:03.053 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:113 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'75, mlcod=14'65)", 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 172.88301819, 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0061123990000000001, 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137061+0000", 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137060+0000", 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137071+0000", 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0706e-05 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137091+0000", 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0052999999999999e-05 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137111+0000", 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9786999999999999e-05 2024-09-13T05:21:03.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139373+0000", 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022619089999999999 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.139430+0000", 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.6953000000000002e-05 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143105+0000", 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0036753519999999998 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143174+0000", 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669000000000003e-05 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'65)", 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:21:03.058 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 172.882508304, 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0056343629999999999, 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137571+0000", 2024-09-13T05:21:03.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137570+0000", 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137583+0000", 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3236e-05 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137610+0000", 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6803000000000001e-05 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.137625+0000", 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5517000000000001e-05 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140029+0000", 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024032580000000001 2024-09-13T05:21:03.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.140060+0000", 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0806000000000003e-05 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143177+0000", 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031170500000000001 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:18:10.143206+0000", 2024-09-13T05:21:03.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9062999999999999e-05 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:46 2.0 e46/16 2:13a6e937:ns::rbd_data.11db3aaeebba.0000000000000008:head v 46'7, mlcod=46'4)", 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 101.647007776, 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.016460243999999999, 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373072+0000", 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373070+0000", 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378383+0000", 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053126279999999998 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378407+0000", 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.4924000000000001e-05 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378421+0000", 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3985000000000001e-05 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378523+0000", 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101989 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.378585+0000", 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1653000000000001e-05 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389448+0000", 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010863413000000001 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.389532+0000", 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3545000000000001e-05 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:51 2.4 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 101.63638743200001, 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.030152937000000001, 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.067 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 51 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383692+0000", 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383690+0000", 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390813+0000", 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0071236190000000003 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390855+0000", 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1267000000000003e-05 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390877+0000", 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2149000000000001e-05 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390963+0000", 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6502000000000006e-05 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391219+0000", 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000256056 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391433+0000", 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000213847 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411744+0000", 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020311330999999998 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413669+0000", 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00192476 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413744+0000", 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4436000000000004e-05 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413845+0000", 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000101396 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.072 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:52 2.4 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 101.629173118, 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.023513338000000002, 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:03.073 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 52 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390906+0000", 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.390904+0000", 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397197+0000", 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0062930809999999998 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397227+0000", 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9748e-05 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397241+0000", 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4185999999999999e-05 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398082+0000", 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00084051600000000003 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398309+0000", 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022787299999999999 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398522+0000", 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021280999999999999 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414110+0000", 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015587790000000001 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414132+0000", 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1895999999999998e-05 2024-09-13T05:21:03.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414295+0000", 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016328999999999999 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414420+0000", 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012455800000000001 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:53 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 101.622817598, 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017425415999999999, 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 53 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.079 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397262+0000", 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397260+0000", 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:03.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397272+0000", 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1163e-05 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397288+0000", 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6583e-05 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397302+0000", 2024-09-13T05:21:03.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3366e-05 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405500+0000", 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081980430000000003 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.405760+0000", 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026009299999999998 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:21:03.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406773+0000", 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0010130040000000001 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414457+0000", 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0076842680000000002 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414569+0000", 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112235 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414648+0000", 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9098999999999996e-05 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414687+0000", 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9062e-05 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:54 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:03.084 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 101.622761234, 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017621616999999999, 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 54 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397318+0000", 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397317+0000", 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397326+0000", 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4619999999999996e-06 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397342+0000", 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5843000000000001e-05 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397348+0000", 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.7159999999999997e-06 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407738+0000", 2024-09-13T05:21:03.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010389419 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408006+0000", 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00026861700000000002 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408957+0000", 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00095102900000000005 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414469+0000", 2024-09-13T05:21:03.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055115720000000002 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414771+0000", 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00030191899999999999 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414849+0000", 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8125000000000002e-05 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.089 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414940+0000", 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0975000000000002e-05 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:55 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 101.622723663, 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017750358000000001, 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:03.090 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 55 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397356+0000", 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397355+0000", 2024-09-13T05:21:03.091 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397363+0000", 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2730000000000002e-06 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397377+0000", 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4102e-05 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.092 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397384+0000", 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.8669999999999996e-06 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409696+0000", 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012311668 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409923+0000", 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00022758000000000001 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.093 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410895+0000", 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00097189900000000003 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414479+0000", 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035840680000000001 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415023+0000", 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00054370500000000001 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.094 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415072+0000", 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9129999999999999e-05 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415106+0000", 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4096e-05 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:56 2.4 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:03.095 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 101.622689139, 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.017931285000000002, 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 56 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:21:03.096 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397390+0000", 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999995 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.097 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397396+0000", 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.1700000000000002e-06 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397418+0000", 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2215999999999999e-05 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.397434+0000", 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5835e-05 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.098 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411827+0000", 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.014392623 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412030+0000", 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00020318499999999999 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 3", 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412625+0000", 2024-09-13T05:21:03.099 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000595135 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.414966+0000", 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023407889999999998 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415180+0000", 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021404 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415264+0000", 2024-09-13T05:21:03.100 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.4035000000000004e-05 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.415322+0000", 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7896999999999998e-05 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.3.0:33 2.1 e105/90 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:7 v 105'33, mlcod=103'29)", 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.081383+0000", 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 4.938696706, 2024-09-13T05:21:03.101 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093908770000000006, 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081383+0000", 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081383+0000", 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.102 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081381+0000", 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081394+0000", 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3166000000000001e-05 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.082330+0000", 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00093517299999999997 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.103 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.082358+0000", 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8232999999999999e-05 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.082498+0000", 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013975000000000001 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.082556+0000", 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8182e-05 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090681+0000", 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0081256049999999993 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090774+0000", 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2387999999999997e-05 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.2.0:81 2.0 e105/16 2:0158b2c3:::rbd_data.11d2ef348a4.000000000000000e:7 v 105'78, mlcod=104'74)", 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.087775+0000", 2024-09-13T05:21:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 4.9323041229999998, 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012558712, 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087775+0000", 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087775+0000", 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.106 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087775+0000", 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087789+0000", 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4443e-05 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087816+0000", 2024-09-13T05:21:03.107 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6630000000000001e-05 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087835+0000", 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9803e-05 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087932+0000", 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.6815000000000001e-05 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.108 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087989+0000", 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.7161999999999999e-05 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.100239+0000", 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012249200999999999 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.100334+0000", 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5557999999999995e-05 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.109 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.2.0:83 2.0 e105/16 2:0d926ee4:::rbd_data.11d2ef348a4.000000000000000d:7 v 105'80, mlcod=104'74)", 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.093447+0000", 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 4.9266326119999997, 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.011226963, 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.093447+0000", 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.093447+0000", 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.093446+0000", 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.093460+0000", 2024-09-13T05:21:03.111 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3817e-05 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.093482+0000", 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2631999999999999e-05 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.093498+0000", 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5486999999999999e-05 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.112 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.093629+0000", 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000131204 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.093707+0000", 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.8127999999999995e-05 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.104589+0000", 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010881617 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.113 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.104674+0000", 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5265000000000004e-05 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.2.0:86 2.0 e105/16 2:1f8e89c5:::rbd_data.11d2ef348a4.0000000000000003:7 v 105'82, mlcod=105'78)", 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.105557+0000", 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 4.9145221719999999, 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055489390000000001, 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.114 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105557+0000", 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105557+0000", 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.115 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105555+0000", 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105575+0000", 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0040000000000001e-05 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105609+0000", 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3812000000000001e-05 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.116 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105628+0000", 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8583000000000001e-05 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105749+0000", 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000121335 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105809+0000", 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9762000000000001e-05 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.111027+0000", 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052176519999999997 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.111106+0000", 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.9762e-05 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.118 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:21:03.120 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T05:21:03.123 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T05:21:03.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T05:21:03.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-13T05:21:03.246 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:21:03.246 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:21:03.247 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:21:03.247 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:21:03.260 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T05:21:03.377 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:21:03.377 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:21:03.377 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:21:03.377 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:21:03.377 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:21:03.391 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4571.0:40 2.1 2:8fdaf644:ns::rbd_data.11db3aaeebba.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 102.171797497, 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.036973964999999998, 2024-09-13T05:21:03.514 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4571", 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/106628764", 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 40 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.515 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340854+0000", 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.340851+0000", 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351774+0000", 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010923179 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351811+0000", 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6641999999999999e-05 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.351837+0000", 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6027999999999999e-05 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.517 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352002+0000", 2024-09-13T05:21:03.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00016454999999999999 2024-09-13T05:21:03.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.518 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.518 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.518 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352400+0000", 2024-09-13T05:21:03.518 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000398649 2024-09-13T05:21:03.518 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.352798+0000", 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00039790300000000002 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.373368+0000", 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.020570060000000001 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.520 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377634+0000", 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0042653379999999996 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377725+0000", 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1064999999999996e-05 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.377828+0000", 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010353699999999999 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.521 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:42 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'14, mlcod=46'11)", 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 102.153973196, 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019515450000000001, 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:21:03.522 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358678+0000", 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358676+0000", 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.523 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358691+0000", 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5642999999999999e-05 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358718+0000", 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6168000000000001e-05 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358736+0000", 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8212000000000001e-05 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.524 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358844+0000", 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010780099999999999 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.358902+0000", 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8189000000000001e-05 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360532+0000", 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0016299750000000001 2024-09-13T05:21:03.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.360630+0000", 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8244000000000003e-05 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:47 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'16, mlcod=46'14)", 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 102.146583455, 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024041309999999999, 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.526 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366068+0000", 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366066+0000", 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:03.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366084+0000", 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8352999999999999e-05 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366117+0000", 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2721000000000001e-05 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366136+0000", 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9049e-05 2024-09-13T05:21:03.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366259+0000", 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000122669 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.366294+0000", 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4788e-05 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.529 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368378+0000", 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0020838139999999998 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.368472+0000", 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4823999999999999e-05 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.530 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:50 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'18, mlcod=46'15)", 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 102.130631182, 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001771314, 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.531 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382021+0000", 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382019+0000", 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382034+0000", 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5201999999999999e-05 2024-09-13T05:21:03.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382064+0000", 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0187000000000001e-05 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382082+0000", 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8084000000000002e-05 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.533 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382197+0000", 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000115333 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.382260+0000", 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.2665999999999999e-05 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383691+0000", 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0014313259999999999 2024-09-13T05:21:03.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.383792+0000", 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100327 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:51 2.4 e46/29 2:24d02f65:ns::rbd_data.11db3aaeebba.000000000000000a:head v 46'19, mlcod=46'17)", 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 102.12099098, 2024-09-13T05:21:03.535 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.018947986999999999, 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391661+0000", 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.391658+0000", 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394384+0000", 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027255449999999998 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.537 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394428+0000", 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4651000000000003e-05 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394452+0000", 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3504000000000001e-05 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394537+0000", 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.5215000000000003e-05 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.538 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.394600+0000", 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.3208999999999997e-05 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410522+0000", 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.015921803000000002 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.410609+0000", 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.6791000000000003e-05 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.539 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:52 2.4 e46/29 2:2a1072dc:ns::rbd_data.11db3aaeebba.000000000000000b:head v 46'20, mlcod=46'17)", 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 102.113898175, 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.013973660000000001, 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398754+0000", 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.398751+0000", 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403105+0000", 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043538780000000003 2024-09-13T05:21:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403136+0000", 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0993e-05 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403156+0000", 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0092999999999999e-05 2024-09-13T05:21:03.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403263+0000", 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106562 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.403312+0000", 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9017000000000002e-05 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.543 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412626+0000", 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0093142740000000009 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.412727+0000", 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000100917 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:53 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'21, mlcod=46'17)", 2024-09-13T05:21:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 102.10568021900001, 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0068888910000000003, 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406971+0000", 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406969+0000", 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.406990+0000", 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0936000000000001e-05 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407024+0000", 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4588000000000002e-05 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.407044+0000", 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9678999999999999e-05 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408880+0000", 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001836124 2024-09-13T05:21:03.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.408917+0000", 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6334999999999997e-05 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413779+0000", 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048627100000000001 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413860+0000", 2024-09-13T05:21:03.548 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.1038999999999994e-05 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:54 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'22, mlcod=46'17)", 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 102.10353440599999, 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0047888590000000003, 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409117+0000", 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409116+0000", 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.550 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409128+0000", 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2259000000000001e-05 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409149+0000", 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1256000000000001e-05 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409161+0000", 2024-09-13T05:21:03.551 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1131999999999999e-05 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409930+0000", 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00076943400000000005 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.409968+0000", 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7666999999999997e-05 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.552 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413864+0000", 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003896044 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413906+0000", 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.2469999999999998e-05 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.4571.0:55 2.4 e46/29 2:2702f9f8:ns::rbd_object_map.11db3aaeebba:head v 46'23, mlcod=46'17)", 2024-09-13T05:21:03.553 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 102.101616612, 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002895459, 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411035+0000", 2024-09-13T05:21:03.554 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411034+0000", 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411044+0000", 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0818999999999999e-05 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411066+0000", 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2022e-05 2024-09-13T05:21:03.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411078+0000", 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1189000000000001e-05 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411161+0000", 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3695000000000004e-05 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.411233+0000", 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2143000000000004e-05 2024-09-13T05:21:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413910+0000", 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026768819999999998 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:21.413931+0000", 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0271999999999998e-05 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:294 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:21:03.557 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.948170349999998, 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018515739999999999, 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 294 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:21:03.558 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564481+0000", 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564485+0000", 2024-09-13T05:21:03.559 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.33e-06 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564497+0000", 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2289e-05 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.564506+0000", 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.2599999999999994e-06 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.560 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565329+0000", 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082232699999999998 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.565462+0000", 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013371300000000001 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.566333+0000", 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000870475 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:296 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.94173241, 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022330969999999999, 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 296 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.562 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570919+0000", 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570918+0000", 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.563 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570923+0000", 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3499999999999999e-06 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570932+0000", 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7019999999999996e-06 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.570940+0000", 2024-09-13T05:21:03.564 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5100000000000001e-06 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572240+0000", 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013003789999999999 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.572365+0000", 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012434699999999999 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.565 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.573152+0000", 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00078769199999999999 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4679.0:304 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 95.928120703999994, 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002027395, 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.566 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4679", 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/1511243783", 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 304 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584531+0000", 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584530+0000", 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584535+0000", 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6770000000000003e-06 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584545+0000", 2024-09-13T05:21:03.568 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.893e-06 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.584552+0000", 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4619999999999999e-06 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.585899+0000", 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0013470839999999999 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.569 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586020+0000", 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012119400000000001 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:27.586558+0000", 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00053795500000000001 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:292 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:21:03.570 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 87.428819593, 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0035814739999999999, 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 292 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083832+0000", 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083829+0000", 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083842+0000", 2024-09-13T05:21:03.572 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.277e-05 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083866+0000", 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3422999999999999e-05 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083882+0000", 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6299000000000001e-05 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.573 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083998+0000", 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000116125 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084009+0000", 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0903e-05 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087241+0000", 2024-09-13T05:21:03.574 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0032323510000000001 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087414+0000", 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000172373 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5052.0:293 2.1 2:86001464:::rbd_data.11d2ef348a4.0000000000000007:head [sparse-read 4096~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-13T05:21:03.575 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 87.428750772000001, 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0038344640000000001, 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5052", 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/4073438582", 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 293 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:21:03.576 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083901+0000", 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083900+0000", 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.577 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083905+0000", 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6539999999999998e-06 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083914+0000", 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.5829999999999996e-06 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.083922+0000", 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.6290000000000001e-06 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.578 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084126+0000", 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000204069 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting_for_map not empty", 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.084143+0000", 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6651999999999999e-05 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087468+0000", 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003325162 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.579 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:36.087735+0000", 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000267569 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4883.0:24 2.1 2:97aa0eeb:::rbd_object_map.11d2ef348a4.0000000000000004:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 83.400348202000004, 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004609854, 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.580 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4883", 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/846758646", 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 24 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112303+0000", 2024-09-13T05:21:03.581 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112301+0000", 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112314+0000", 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2856e-05 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112337+0000", 2024-09-13T05:21:03.582 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2889999999999999e-05 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112353+0000", 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6452000000000001e-05 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112445+0000", 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.1910000000000006e-05 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op must wait for map", 2024-09-13T05:21:03.583 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.112453+0000", 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4749999999999996e-06 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.114940+0000", 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002486977 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115052+0000", 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000112803 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:03.584 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.115158+0000", 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105512 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116033+0000", 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00087509800000000004 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116861+0000", 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00082842199999999999 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.585 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116893+0000", 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1275000000000001e-05 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:40.116913+0000", 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0588e-05 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5382.0:50 3.2 3:52b68663:::rbd_header.14f41729d63b:head [watch unwatch cookie 140129079413376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:21:03.586 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 74.873667888, 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018329869999999999, 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5382", 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/2117079999", 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 50 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638984+0000", 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638982+0000", 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.638994+0000", 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.2466000000000001e-05 2024-09-13T05:21:03.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639025+0000", 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1158999999999998e-05 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639040+0000", 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.4756999999999999e-05 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639139+0000", 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.9049999999999995e-05 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.589 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639268+0000", 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012908 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 2", 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.639487+0000", 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000218829 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640456+0000", 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00096863699999999995 2024-09-13T05:21:03.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640651+0000", 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019548100000000001 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640726+0000", 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5451000000000007e-05 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:48.640817+0000", 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.0294e-05 2024-09-13T05:21:03.591 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5603.0:28 2.1 2:96923c38:::journal_data.2.11d2ef348a4.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e95)", 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:48.266800+0000", 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 15.245852162, 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027118490000000001, 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5603", 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/389233024", 2024-09-13T05:21:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 28 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266800+0000", 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266800+0000", 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.593 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266797+0000", 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266811+0000", 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3876999999999999e-05 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266840+0000", 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8909000000000001e-05 2024-09-13T05:21:03.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266860+0000", 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.0089999999999999e-05 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.266999+0000", 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013951999999999999 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.267232+0000", 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000232434 2024-09-13T05:21:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for subops from 5", 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.267506+0000", 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000274228 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "op_commit", 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.268630+0000", 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0011239150000000001 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "sub_op_commit_rec", 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.269382+0000", 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00075244899999999998 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.596 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.269458+0000", 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.5984999999999998e-05 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:48.269511+0000", 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3189000000000002e-05 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.5618.0:164 2.1 2:96923c38:::journal_data.2.11d2ef348a4.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e96)", 2024-09-13T05:21:03.597 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:49.195437+0000", 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 14.317215179, 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0030533930000000002, 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.5618", 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/919220950", 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 164 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195437+0000", 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.598 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195437+0000", 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195434+0000", 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999971 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.599 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195448+0000", 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3816e-05 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195480+0000", 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2243000000000003e-05 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195498+0000", 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7995999999999998e-05 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.600 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195645+0000", 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00014696699999999999 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.195777+0000", 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00013206399999999999 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:49.198490+0000", 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0027129599999999999 2024-09-13T05:21:03.601 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.5.0:76 2.4 e105/90 2:30e271b9:::rbd_data.11d2ef348a4.000000000000000b:7 v 105'79, mlcod=104'75)", 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.089079+0000", 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.4235728720000003, 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00535726, 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.602 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.089079+0000", 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.089079+0000", 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.089076+0000", 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.603 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.089095+0000", 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.808e-05 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.089138+0000", 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.3645e-05 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.089158+0000", 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.9904000000000001e-05 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.604 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.089347+0000", 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018898600000000001 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.089403+0000", 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5788999999999997e-05 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.094349+0000", 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049458729999999999 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.605 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.094436+0000", 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.7359000000000002e-05 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.5.0:77 2.4 e105/90 2:3b031e57:::rbd_data.11d2ef348a4.000000000000000c:7 v 105'81, mlcod=104'75)", 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.094889+0000", 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.4177623009999998, 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0069722229999999996, 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:03.606 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.094889+0000", 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.094889+0000", 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.094888+0000", 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:03.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.094904+0000", 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6650000000000002e-05 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.094937+0000", 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2944999999999997e-05 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.094956+0000", 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8539999999999999e-05 2024-09-13T05:21:03.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.095059+0000", 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00010343799999999999 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.095115+0000", 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.5628999999999999e-05 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.101745+0000", 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0066297630000000003 2024-09-13T05:21:03.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.101862+0000", 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00011692399999999999 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:03.610 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:21:03.611 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T05:21:03.739 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:21:03.739 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:21:03.739 INFO:teuthology.orchestra.run.smithi079.stdout: "num_ops": 0 2024-09-13T05:21:03.739 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:21:03.753 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T05:21:03.871 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:21:03.872 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [], 2024-09-13T05:21:03.872 INFO:teuthology.orchestra.run.smithi079.stdout: "complaint_time": 30, 2024-09-13T05:21:03.872 INFO:teuthology.orchestra.run.smithi079.stdout: "num_blocked_ops": 0 2024-09-13T05:21:03.872 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:21:03.885 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T05:21:04.006 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-13T05:21:04.006 INFO:teuthology.orchestra.run.smithi079.stdout: "size": 20, 2024-09-13T05:21:04.006 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 600, 2024-09-13T05:21:04.006 INFO:teuthology.orchestra.run.smithi079.stdout: "ops": [ 2024-09-13T05:21:04.006 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_op(client.4787.0:12 2.6 2:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_get in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.958278464000003, 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032917566000000002, 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "client_info": { 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "client": "client.4787", 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "client_addr": "172.21.15.122:0/3910878498", 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "tid": 12 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.007 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045395+0000", 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045392+0000", 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.008 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045406+0000", 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3643e-05 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045429+0000", 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.3008999999999999e-05 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045445+0000", 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6229e-05 2024-09-13T05:21:04.009 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045565+0000", 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120035 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "waiting for peered", 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.045587+0000", 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1365999999999998e-05 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078059+0000", 2024-09-13T05:21:04.010 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.032472398 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.078312+0000", 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00025307599999999997 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.921877886000004, 2024-09-13T05:21:04.011 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.005844592, 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081795+0000", 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.012 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.081794+0000", 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084037+0000", 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022428830000000002 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084072+0000", 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4824999999999999e-05 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.013 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084119+0000", 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.7383999999999999e-05 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084245+0000", 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000126119 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.084266+0000", 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1115e-05 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.014 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087640+0000", 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0033737089999999999 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.918294582000001, 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0058523960000000002, 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.015 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085379+0000", 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.085377+0000", 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087736+0000", 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023585780000000001 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087782+0000", 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.6128000000000003e-05 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.087813+0000", 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0905000000000003e-05 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.017 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088003+0000", 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000190552 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.088045+0000", 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1922000000000003e-05 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091231+0000", 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003185551 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.018 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head, version: 46'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a1bc2ec8:ns::rbd_data.11db3aaeebba.0000000000000002:head@46'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.913660496000006, 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0052072630000000002, 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.019 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090013+0000", 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.090011+0000", 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091860+0000", 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018486799999999999 2024-09-13T05:21:04.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091894+0000", 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.3531999999999997e-05 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.091920+0000", 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6265999999999999e-05 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092047+0000", 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00012725799999999999 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.092066+0000", 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8776000000000001e-05 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095220+0000", 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003153931 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head, version: 46'7, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:64fb33e6:ns::rbd_data.11db3aaeebba.0000000000000009:head@46'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:04.022 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.910423219999998, 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0050151600000000003, 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093250+0000", 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.093248+0000", 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095418+0000", 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0021695159999999998 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095453+0000", 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5505e-05 2024-09-13T05:21:04.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095492+0000", 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.9295000000000002e-05 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095731+0000", 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00023887299999999999 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.095773+0000", 2024-09-13T05:21:04.025 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1862e-05 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.098265+0000", 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0024918890000000002 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:21:04.026 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.886745242000003, 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0044505109999999999, 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116928+0000", 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.027 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.116926+0000", 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118855+0000", 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001928953 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118888+0000", 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2659e-05 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.028 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.118915+0000", 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7019e-05 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119106+0000", 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00019064799999999999 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.119143+0000", 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.6962000000000003e-05 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.029 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.121378+0000", 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0022358769999999998 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.882903282000001, 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043087560000000004, 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.030 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120770+0000", 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.031 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.120768+0000", 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122846+0000", 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00207843 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122888+0000", 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.1548999999999998e-05 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.032 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.122916+0000", 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8082e-05 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123105+0000", 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018852499999999999 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.123143+0000", 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.8294999999999998e-05 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.033 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.125079+0000", 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001935672 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.5 55/54 [PushOp(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head, version: 46'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2725e42:ns::rbd_data.11db3aaeebba.0000000000000004:head@46'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.879067886000001, 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0049026850000000004, 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.034 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124605+0000", 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.124603+0000", 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:04.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126937+0000", 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002334109 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.126973+0000", 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5858000000000001e-05 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127002+0000", 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.8496e-05 2024-09-13T05:21:04.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127189+0000", 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018715499999999999 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.127226+0000", 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7246000000000002e-05 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.129508+0000", 2024-09-13T05:21:04.037 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.002281787 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head, version: 46'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:74be0f09:ns::rbd_data.11db3aaeebba.000000000000000c:head@46'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.868459870999999, 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0045497360000000004, 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.038 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135213+0000", 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.135212+0000", 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.039 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137070+0000", 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001858305 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137116+0000", 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.5791999999999998e-05 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137188+0000", 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.2349999999999997e-05 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137362+0000", 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017399000000000001 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.137398+0000", 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5808e-05 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.139763+0000", 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0023648710000000002 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.860340852999997, 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048203120000000002, 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143332+0000", 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.143331+0000", 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.043 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145237+0000", 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0019056310000000001 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145268+0000", 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.1652000000000001e-05 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145294+0000", 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5477999999999999e-05 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.044 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145475+0000", 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00018084000000000001 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.145510+0000", 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.5250999999999998e-05 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.148153+0000", 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0026428070000000001 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.045 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "MOSDPGPush(2.6 55/54 [PushOp(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head, version: 46'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7cf61e77:ns::rbd_data.11db3aaeebba.000000000000000d:head@46'9, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 88.845756414999997, 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0048846460000000003, 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "started", 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157917+0000", 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.157915+0000", 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.047 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159238+0000", 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001323863 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159320+0000", 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.2004999999999998e-05 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159346+0000", 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.5939e-05 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.048 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159590+0000", 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00024374600000000001 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.159628+0000", 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.7962e-05 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:19:35.162801+0000", 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0031732840000000002 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.049 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "pg_update_log_missing(2.5 epoch 104/98 rep_tid 120 entries 104'60 (0'0) error 2:a1fa0702:::rbd_object_map.11d2ef348a4.0000000000000008:head by client.6243.0:53 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 104'57)", 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:57.090489+0000", 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 6.9131837530000002, 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0054663740000000004, 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "reached pg", 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090489+0000", 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090489+0000", 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090489+0000", 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090499+0000", 2024-09-13T05:21:04.051 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0457e-05 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090515+0000", 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6178999999999999e-05 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090525+0000", 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4080000000000004e-06 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.052 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.090574+0000", 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.9675999999999997e-05 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:57.095956+0000", 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0053814819999999999 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.053 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.0.0:123 2.7 e105/100 2:e246a13f:::rbd_data.11d2ef348a4.0000000000000008:7 v 105'232, mlcod=104'228)", 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.081316+0000", 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.9223567050000003, 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010995694, 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081316+0000", 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.054 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081316+0000", 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081315+0000", 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081332+0000", 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.6503000000000001e-05 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081364+0000", 2024-09-13T05:21:04.055 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.2175999999999998e-05 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081382+0000", 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7915999999999999e-05 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081594+0000", 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021225899999999999 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.056 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.081669+0000", 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.4685999999999996e-05 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.091262+0000", 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0095934130000000003 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.092312+0000", 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.001049714 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.0.0:125 2.7 e105/100 2:f5d87c9f:::rbd_data.11d2ef348a4.0000000000000000:7 v 105'234, mlcod=104'228)", 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.084770+0000", 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.9189034469999999, 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.012268021, 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.084770+0000", 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.058 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.084770+0000", 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.084767+0000", 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.084786+0000", 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.8825999999999999e-05 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.059 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.084816+0000", 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0029999999999999e-05 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.084839+0000", 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.2532e-05 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.086699+0000", 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0018602619999999999 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.060 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.086758+0000", 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.9095999999999998e-05 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.096949+0000", 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010190696000000001 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.097038+0000", 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.9124999999999998e-05 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.6243.0:56 2.3 e105/100 2:d073633c:::rbd_header.11d2ef348a4:head v 105'360, mlcod=104'358)", 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.087883+0000", 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.9157905509999997, 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0084321450000000003, 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087883+0000", 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.062 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087883+0000", 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087882+0000", 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.999999 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.063 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087893+0000", 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.1533e-05 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087923+0000", 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.9709999999999998e-05 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.087940+0000", 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7482e-05 2024-09-13T05:21:04.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.088116+0000", 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00017586699999999999 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.088170+0000", 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.3409000000000003e-05 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.096221+0000", 2024-09-13T05:21:04.065 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0080510890000000009 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.096315+0000", 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.4122000000000004e-05 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.2.0:84 2.3 e105/100 2:df9bac5a:::rbd_data.11d2ef348a4.000000000000000f:7 v 105'362, mlcod=104'358)", 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.090412+0000", 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.9132607349999997, 2024-09-13T05:21:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0075842510000000002, 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090412+0000", 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090412+0000", 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.067 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090410+0000", 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090428+0000", 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7520000000000002e-05 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090455+0000", 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.7521999999999998e-05 2024-09-13T05:21:04.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090473+0000", 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7940000000000001e-05 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090634+0000", 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000160724 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.090689+0000", 2024-09-13T05:21:04.069 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.4809000000000003e-05 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.097783+0000", 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0070939870000000004 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.097997+0000", 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.00021383599999999999 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.0.0:127 2.7 e105/100 2:ff4c2c0f:::rbd_data.11d2ef348a4.0000000000000009:7 v 105'236, mlcod=105'232)", 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.097991+0000", 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.9056817410000004, 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055439829999999997, 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.097991+0000", 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.071 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.097991+0000", 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.097989+0000", 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098034+0000", 2024-09-13T05:21:04.072 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4.4736e-05 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098069+0000", 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.4347000000000002e-05 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098082+0000", 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3202e-05 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.073 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098179+0000", 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7065000000000007e-05 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098245+0000", 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 6.6668999999999995e-05 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.103436+0000", 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0051910430000000002 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.103535+0000", 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.8870999999999996e-05 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.0.0:128 2.5 e105/98 2:ad1d6908:::rbd_data.11d2ef348a4.0000000000000004:7 v 105'63, mlcod=104'60)", 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.098104+0000", 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.905568873, 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.010102174, 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.075 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098104+0000", 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098104+0000", 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098103+0000", 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999986 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.076 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098113+0000", 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.0336000000000001e-05 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098131+0000", 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7886000000000001e-05 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.077 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098145+0000", 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.3703000000000001e-05 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098251+0000", 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000106708 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.098323+0000", 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 7.1216000000000004e-05 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.078 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.108113+0000", 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0097901959999999993 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.108206+0000", 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.3604999999999996e-05 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(osd.0.0:129 2.5 e105/98 2:b18cf895:::rbd_data.11d2ef348a4.0000000000000002:7 v 105'65, mlcod=104'60)", 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.099426+0000", 2024-09-13T05:21:04.079 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.9042468909999997, 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0095346010000000002, 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.099426+0000", 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.099426+0000", 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.080 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.099424+0000", 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999981 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.099440+0000", 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.5277e-05 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.099467+0000", 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.6954999999999999e-05 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.081 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.099488+0000", 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1537000000000001e-05 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.103294+0000", 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.003806112 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.103353+0000", 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 5.8412000000000002e-05 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.082 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.108864+0000", 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0055110530000000001 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.108961+0000", 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 9.7104999999999999e-05 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.083 INFO:teuthology.orchestra.run.smithi079.stdout: "description": "osd_repop(client.6243.0:65 2.3 e105/100 2:d073633c:::rbd_header.11d2ef348a4:head v 105'363, mlcod=105'360)", 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "initiated_at": "2024-09-13T05:20:58.104937+0000", 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 5.8987364969999998, 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.004709815, 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "type_data": { 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "events": [ 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "initiated", 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.104937+0000", 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "throttled", 2024-09-13T05:21:04.084 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.104937+0000", 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "header_read", 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.104934+0000", 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 4294967295.9999976 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "all_read", 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.104952+0000", 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 1.7927e-05 2024-09-13T05:21:04.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "dispatched", 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.104982+0000", 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 3.0022000000000002e-05 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "queued_for_pg", 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105004+0000", 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 2.1494000000000001e-05 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "reached_pg", 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105124+0000", 2024-09-13T05:21:04.086 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000120331 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "started", 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.105207+0000", 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 8.3300999999999994e-05 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "commit_sent", 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.109541+0000", 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.0043338220000000002 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-13T05:21:04.087 INFO:teuthology.orchestra.run.smithi079.stdout: "event": "done", 2024-09-13T05:21:04.088 INFO:teuthology.orchestra.run.smithi079.stdout: "time": "2024-09-13T05:20:58.109646+0000", 2024-09-13T05:21:04.088 INFO:teuthology.orchestra.run.smithi079.stdout: "duration": 0.000105262 2024-09-13T05:21:04.088 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.088 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.088 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.088 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-13T05:21:04.088 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-13T05:21:04.088 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-13T05:21:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-13T05:21:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T05:21:04.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T05:21:04.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T05:21:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T05:21:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-13T05:21:05.172 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-13T05:21:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T05:21:06.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T05:21:07.039 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T05:21:07.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-13T05:21:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T05:21:08.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T05:21:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T05:21:09.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-13T05:21:10.154 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T05:21:10.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T05:21:11.163 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T05:21:11.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-13T05:21:12.172 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T05:21:12.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T05:21:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T05:21:13.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-13T05:21:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T05:21:14.212 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T05:21:14.213 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T05:21:14.213 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T05:21:14.213 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T05:21:14.213 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T05:21:14.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T05:21:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:14.517 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":121,"fsid":"1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5","created":"2024-09-13T05:17:57.363809+0000","modified":"2024-09-13T05:21:14.182493+0000","last_up_change":"2024-09-13T05:19:34.047702+0000","last_in_change":"2024-09-13T05:20:35.926417+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T05:18:08.122164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T05:18:15.001664+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e58174f4-85ec-4321-8bc3-384261c59c6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6801","nonce":2153409514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6803","nonce":2153409514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6807","nonce":2153409514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6805","nonce":2153409514}]},"public_addr":"172.21.15.16:6801/2153409514","cluster_addr":"172.21.15.16:6803/2153409514","heartbeat_back_addr":"172.21.15.16:6807/2153409514","heartbeat_front_addr":"172.21.15.16:6805/2153409514","state":["exists","up"]},{"osd":1,"uuid":"84a3db6c-dacd-43d5-b964-37fdef0091a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6809","nonce":2608485151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6811","nonce":2608485151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6815","nonce":2608485151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6813","nonce":2608485151}]},"public_addr":"172.21.15.16:6809/2608485151","cluster_addr":"172.21.15.16:6811/2608485151","heartbeat_back_addr":"172.21.15.16:6815/2608485151","heartbeat_front_addr":"172.21.15.16:6813/2608485151","state":["exists","up"]},{"osd":2,"uuid":"0d530476-cf38-4544-9ce7-e8ed35df220b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6817","nonce":1006404377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6819","nonce":1006404377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6823","nonce":1006404377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6821","nonce":1006404377}]},"public_addr":"172.21.15.16:6817/1006404377","cluster_addr":"172.21.15.16:6819/1006404377","heartbeat_back_addr":"172.21.15.16:6823/1006404377","heartbeat_front_addr":"172.21.15.16:6821/1006404377","state":["exists","up"]},{"osd":3,"uuid":"fd91f7e2-580d-47ea-9db5-792c8f1ebe01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":12,"last_clean_end":20,"up_from":29,"up_thru":90,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6817","nonce":1834661236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6819","nonce":1834661236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6823","nonce":1834661236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6821","nonce":1834661236}]},"public_addr":"172.21.15.79:6817/1834661236","cluster_addr":"172.21.15.79:6819/1834661236","heartbeat_back_addr":"172.21.15.79:6823/1834661236","heartbeat_front_addr":"172.21.15.79:6821/1834661236","state":["exists","up"]},{"osd":4,"uuid":"01af0f4c-5da2-46fe-a262-59fcc6a2efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":12,"last_clean_end":38,"up_from":54,"up_thru":87,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6809","nonce":3457694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6811","nonce":3457694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6815","nonce":3457694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6813","nonce":3457694380}]},"public_addr":"172.21.15.79:6809/3457694380","cluster_addr":"172.21.15.79:6811/3457694380","heartbeat_back_addr":"172.21.15.79:6815/3457694380","heartbeat_front_addr":"172.21.15.79:6813/3457694380","state":["exists","up"]},{"osd":5,"uuid":"64cdd5f2-1568-4e96-ad90-6ef7414651f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6801","nonce":2778785304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6803","nonce":2778785304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6807","nonce":2778785304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6805","nonce":2778785304}]},"public_addr":"172.21.15.79:6801/2778785304","cluster_addr":"172.21.15.79:6803/2778785304","heartbeat_back_addr":"172.21.15.79:6807/2778785304","heartbeat_front_addr":"172.21.15.79:6805/2778785304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.193869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.255764+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.286354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.221310+0000","dead_epoch":21},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.026235+0000","dead_epoch":40},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.010242+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.2","osds":[0,1]}],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T05:21:14.536 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T05:21:14.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T05:21:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:14.829 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":121,"fsid":"1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5","created":"2024-09-13T05:17:57.363809+0000","modified":"2024-09-13T05:21:14.182493+0000","last_up_change":"2024-09-13T05:19:34.047702+0000","last_in_change":"2024-09-13T05:20:35.926417+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T05:18:08.122164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T05:18:15.001664+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e58174f4-85ec-4321-8bc3-384261c59c6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6801","nonce":2153409514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6803","nonce":2153409514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6807","nonce":2153409514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6805","nonce":2153409514}]},"public_addr":"172.21.15.16:6801/2153409514","cluster_addr":"172.21.15.16:6803/2153409514","heartbeat_back_addr":"172.21.15.16:6807/2153409514","heartbeat_front_addr":"172.21.15.16:6805/2153409514","state":["exists","up"]},{"osd":1,"uuid":"84a3db6c-dacd-43d5-b964-37fdef0091a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6809","nonce":2608485151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6811","nonce":2608485151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6815","nonce":2608485151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6813","nonce":2608485151}]},"public_addr":"172.21.15.16:6809/2608485151","cluster_addr":"172.21.15.16:6811/2608485151","heartbeat_back_addr":"172.21.15.16:6815/2608485151","heartbeat_front_addr":"172.21.15.16:6813/2608485151","state":["exists","up"]},{"osd":2,"uuid":"0d530476-cf38-4544-9ce7-e8ed35df220b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6817","nonce":1006404377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6819","nonce":1006404377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6823","nonce":1006404377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6821","nonce":1006404377}]},"public_addr":"172.21.15.16:6817/1006404377","cluster_addr":"172.21.15.16:6819/1006404377","heartbeat_back_addr":"172.21.15.16:6823/1006404377","heartbeat_front_addr":"172.21.15.16:6821/1006404377","state":["exists","up"]},{"osd":3,"uuid":"fd91f7e2-580d-47ea-9db5-792c8f1ebe01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":12,"last_clean_end":20,"up_from":29,"up_thru":90,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6817","nonce":1834661236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6819","nonce":1834661236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6823","nonce":1834661236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6821","nonce":1834661236}]},"public_addr":"172.21.15.79:6817/1834661236","cluster_addr":"172.21.15.79:6819/1834661236","heartbeat_back_addr":"172.21.15.79:6823/1834661236","heartbeat_front_addr":"172.21.15.79:6821/1834661236","state":["exists","up"]},{"osd":4,"uuid":"01af0f4c-5da2-46fe-a262-59fcc6a2efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":12,"last_clean_end":38,"up_from":54,"up_thru":87,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6809","nonce":3457694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6811","nonce":3457694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6815","nonce":3457694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6813","nonce":3457694380}]},"public_addr":"172.21.15.79:6809/3457694380","cluster_addr":"172.21.15.79:6811/3457694380","heartbeat_back_addr":"172.21.15.79:6815/3457694380","heartbeat_front_addr":"172.21.15.79:6813/3457694380","state":["exists","up"]},{"osd":5,"uuid":"64cdd5f2-1568-4e96-ad90-6ef7414651f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6801","nonce":2778785304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6803","nonce":2778785304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6807","nonce":2778785304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6805","nonce":2778785304}]},"public_addr":"172.21.15.79:6801/2778785304","cluster_addr":"172.21.15.79:6803/2778785304","heartbeat_back_addr":"172.21.15.79:6807/2778785304","heartbeat_front_addr":"172.21.15.79:6805/2778785304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.193869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.255764+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.286354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.221310+0000","dead_epoch":21},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.026235+0000","dead_epoch":40},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.010242+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.2","osds":[0,1]}],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T05:21:14.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T05:21:14.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T05:21:14.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T05:21:14.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T05:21:14.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T05:21:14.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T05:21:15.011 INFO:teuthology.orchestra.run.smithi016.stdout:51539607594 2024-09-13T05:21:15.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T05:21:15.028 INFO:teuthology.orchestra.run.smithi016.stdout:231928234006 2024-09-13T05:21:15.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T05:21:15.072 INFO:teuthology.orchestra.run.smithi016.stdout:51539607595 2024-09-13T05:21:15.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T05:21:15.075 INFO:teuthology.orchestra.run.smithi016.stdout:51539607594 2024-09-13T05:21:15.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T05:21:15.091 INFO:teuthology.orchestra.run.smithi016.stdout:51539607594 2024-09-13T05:21:15.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T05:21:15.097 INFO:teuthology.orchestra.run.smithi016.stdout:124554051616 2024-09-13T05:21:15.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T05:21:15.409 INFO:teuthology.orchestra.run.smithi016.stdout:51539607594 2024-09-13T05:21:15.411 INFO:teuthology.orchestra.run.smithi016.stdout:231928234005 2024-09-13T05:21:15.429 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607595 got 51539607594 for osd.5 2024-09-13T05:21:15.439 INFO:tasks.ceph.ceph_manager.ceph:need seq 231928234006 got 231928234005 for osd.4 2024-09-13T05:21:15.441 INFO:teuthology.orchestra.run.smithi016.stdout:51539607593 2024-09-13T05:21:15.455 INFO:teuthology.orchestra.run.smithi016.stdout:51539607593 2024-09-13T05:21:15.458 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607594 got 51539607593 for osd.2 2024-09-13T05:21:15.479 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607594 got 51539607593 for osd.1 2024-09-13T05:21:15.546 INFO:teuthology.orchestra.run.smithi016.stdout:51539607593 2024-09-13T05:21:15.563 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607594 got 51539607593 for osd.0 2024-09-13T05:21:15.563 INFO:teuthology.orchestra.run.smithi016.stdout:124554051615 2024-09-13T05:21:15.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 124554051616 got 124554051615 for osd.3 2024-09-13T05:21:16.430 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T05:21:16.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T05:21:16.459 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T05:21:16.480 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T05:21:16.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T05:21:16.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T05:21:16.881 INFO:teuthology.orchestra.run.smithi016.stdout:51539607594 2024-09-13T05:21:16.898 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607594 got 51539607594 for osd.2 2024-09-13T05:21:16.899 DEBUG:teuthology.parallel:result is None 2024-09-13T05:21:16.904 INFO:teuthology.orchestra.run.smithi016.stdout:51539607595 2024-09-13T05:21:16.922 INFO:teuthology.orchestra.run.smithi016.stdout:231928234006 2024-09-13T05:21:16.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607595 got 51539607595 for osd.5 2024-09-13T05:21:16.928 DEBUG:teuthology.parallel:result is None 2024-09-13T05:21:16.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 231928234006 got 231928234006 for osd.4 2024-09-13T05:21:16.947 DEBUG:teuthology.parallel:result is None 2024-09-13T05:21:16.965 INFO:teuthology.orchestra.run.smithi016.stdout:51539607594 2024-09-13T05:21:16.969 INFO:teuthology.orchestra.run.smithi016.stdout:124554051616 2024-09-13T05:21:16.984 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607594 got 51539607594 for osd.1 2024-09-13T05:21:16.985 DEBUG:teuthology.parallel:result is None 2024-09-13T05:21:16.987 INFO:tasks.ceph.ceph_manager.ceph:need seq 124554051616 got 124554051616 for osd.3 2024-09-13T05:21:16.987 DEBUG:teuthology.parallel:result is None 2024-09-13T05:21:16.994 INFO:teuthology.orchestra.run.smithi016.stdout:51539607594 2024-09-13T05:21:17.010 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607594 got 51539607594 for osd.0 2024-09-13T05:21:17.011 DEBUG:teuthology.parallel:result is None 2024-09-13T05:21:17.011 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T05:21:17.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:21:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:21:17.346 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-13T05:21:16.096016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8392,"num_read_kb":199260,"num_write":1143,"num_write_kb":275373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":162198480,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1243,"ondisk_log_size":1243,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":202520,"kb_used_data":9532,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":566028520,"statfs":{"total":579820584960,"available":579613204480,"internally_reserved":0,"allocated":9760768,"data_stored":4535618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.935329"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945720+0000","last_change":"2024-09-13T05:21:09.054560+0000","last_active":"2024-09-13T05:21:13.945720+0000","last_peered":"2024-09-13T05:21:13.945720+0000","last_clean":"2024-09-13T05:21:13.945720+0000","last_became_active":"2024-09-13T05:21:09.054143+0000","last_became_peered":"2024-09-13T05:21:09.054143+0000","last_unstale":"2024-09-13T05:21:13.945720+0000","last_undegraded":"2024-09-13T05:21:13.945720+0000","last_fullsized":"2024-09-13T05:21:13.945720+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:53.562792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"107'398","reported_seq":2741,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945778+0000","last_change":"2024-09-13T05:21:09.054383+0000","last_active":"2024-09-13T05:21:13.945778+0000","last_peered":"2024-09-13T05:21:13.945778+0000","last_clean":"2024-09-13T05:21:13.945778+0000","last_became_active":"2024-09-13T05:21:09.053687+0000","last_became_peered":"2024-09-13T05:21:09.053687+0000","last_unstale":"2024-09-13T05:21:13.945778+0000","last_undegraded":"2024-09-13T05:21:13.945778+0000","last_fullsized":"2024-09-13T05:21:13.945778+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:41.805028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070414999999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5324,"num_read_kb":28521,"num_write":392,"num_write_kb":20803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":12648486,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"107'91","reported_seq":436,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.180355+0000","last_change":"2024-09-13T05:21:00.105385+0000","last_active":"2024-09-13T05:21:13.180355+0000","last_peered":"2024-09-13T05:21:13.180355+0000","last_clean":"2024-09-13T05:21:13.180355+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:21:13.180355+0000","last_undegraded":"2024-09-13T05:21:13.180355+0000","last_fullsized":"2024-09-13T05:21:13.180355+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074870500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":27203,"num_write":68,"num_write_kb":49155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"107'36","reported_seq":111,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.181366+0000","last_change":"2024-09-13T05:21:00.107196+0000","last_active":"2024-09-13T05:21:13.181366+0000","last_peered":"2024-09-13T05:21:13.181366+0000","last_clean":"2024-09-13T05:21:13.181366+0000","last_became_active":"2024-09-13T05:20:36.954764+0000","last_became_peered":"2024-09-13T05:20:36.954764+0000","last_unstale":"2024-09-13T05:21:13.181366+0000","last_undegraded":"2024-09-13T05:21:13.181366+0000","last_fullsized":"2024-09-13T05:21:13.181366+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:11:18.106127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054390799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":8215,"num_write":27,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"106'104","reported_seq":565,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.935859+0000","last_change":"2024-09-13T05:21:00.114748+0000","last_active":"2024-09-13T05:21:13.935859+0000","last_peered":"2024-09-13T05:21:13.935859+0000","last_clean":"2024-09-13T05:21:13.935859+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:21:13.935859+0000","last_undegraded":"2024-09-13T05:21:13.935859+0000","last_fullsized":"2024-09-13T05:21:13.935859+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070499600000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":38450,"num_write":79,"num_write_kb":38939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"107'97","reported_seq":403,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.856968+0000","last_change":"2024-09-13T05:21:00.112104+0000","last_active":"2024-09-13T05:21:12.856968+0000","last_peered":"2024-09-13T05:21:12.856968+0000","last_clean":"2024-09-13T05:21:12.856968+0000","last_became_active":"2024-09-13T05:20:36.957689+0000","last_became_peered":"2024-09-13T05:20:36.957689+0000","last_unstale":"2024-09-13T05:21:12.856968+0000","last_undegraded":"2024-09-13T05:21:12.856968+0000","last_fullsized":"2024-09-13T05:21:12.856968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T06:29:41.023217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001858878,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":16522,"num_write":101,"num_write_kb":32810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":33554508,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"107'73","reported_seq":43,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945775+0000","last_change":"2024-09-13T05:21:06.170554+0000","last_active":"2024-09-13T05:21:13.945775+0000","last_peered":"2024-09-13T05:21:13.945775+0000","last_clean":"2024-09-13T05:21:13.945775+0000","last_became_active":"2024-09-13T05:21:06.157892+0000","last_became_peered":"2024-09-13T05:21:06.157892+0000","last_unstale":"2024-09-13T05:21:13.945775+0000","last_undegraded":"2024-09-13T05:21:13.945775+0000","last_fullsized":"2024-09-13T05:21:13.945775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:52.415500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081846100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":19146,"num_write":56,"num_write_kb":41013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":41992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'84","reported_seq":211,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.857059+0000","last_change":"2024-09-13T05:21:00.112146+0000","last_active":"2024-09-13T05:21:12.857059+0000","last_peered":"2024-09-13T05:21:12.857059+0000","last_clean":"2024-09-13T05:21:12.857059+0000","last_became_active":"2024-09-13T05:20:43.986894+0000","last_became_peered":"2024-09-13T05:20:43.986894+0000","last_unstale":"2024-09-13T05:21:12.857059+0000","last_undegraded":"2024-09-13T05:21:12.857059+0000","last_fullsized":"2024-09-13T05:21:12.857059+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:15:48.175749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019911320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":516,"num_read_kb":12708,"num_write":57,"num_write_kb":24588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29360244,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"107'243","reported_seq":1609,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945824+0000","last_change":"2024-09-13T05:21:09.052792+0000","last_active":"2024-09-13T05:21:13.945824+0000","last_peered":"2024-09-13T05:21:13.945824+0000","last_clean":"2024-09-13T05:21:13.945824+0000","last_became_active":"2024-09-13T05:21:09.051755+0000","last_became_peered":"2024-09-13T05:21:09.051755+0000","last_unstale":"2024-09-13T05:21:13.945824+0000","last_undegraded":"2024-09-13T05:21:13.945824+0000","last_fullsized":"2024-09-13T05:21:13.945824+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:32.347051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291826,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1348,"num_read_kb":48395,"num_write":204,"num_write_kb":49262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":23080975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8276,"num_read_kb":199160,"num_write":984,"num_write_kb":272957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":161608112,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1126,"ondisk_log_size":1126,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607595,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":2007040,"data_stored":1144100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.071,"15min":1.071},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.89700000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51900000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.028}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0580000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Sep 13 05:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.997}]}]},{"osd":4,"up_from":54,"seq":231928234006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29488,"kb_used_data":1380,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342352,"statfs":{"total":96636764160,"available":96606568448,"internally_reserved":0,"allocated":1413120,"data_stored":553732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.95199999999999996}]},{"osd":1,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.83899999999999997}]},{"osd":2,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.78700000000000003}]},{"osd":3,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.86499999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.753},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.92900000000000005}]}]},{"osd":3,"up_from":29,"seq":124554051616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":1.246},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69599999999999995,"15min":0.69999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":1.222}]},{"osd":1,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68300000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.627}]},{"osd":2,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67800000000000005,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92000000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.53200000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.96699999999999997}]},{"osd":4,"last update":"Fri Sep 13 05:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.1779999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68300000000000005,"15min":0.66800000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.105},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.89400000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36608,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1425408,"data_stored":558368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.28299999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.95899999999999996,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.59099999999999997},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.083,"5min":1.171,"15min":1.171},"last":1.083},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86099999999999999,"5min":1.21,"15min":1.21},"last":0.755}]},{"osd":3,"last update":"Fri Sep 13 05:20:45 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.90900000000000003,"5min":1.083,"15min":1.083},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.751,"15min":0.752},"min":{"1min":0.54700000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93700000000000006,"5min":1.21,"15min":1.21},"last":1.028}]},{"osd":4,"last update":"Fri Sep 13 05:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":1.206}]},{"osd":5,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3799999999999999,"5min":1.544,"15min":1.544},"last":1.3089999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.1399999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607594,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":2035712,"data_stored":1153391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88800000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":1.119}]},{"osd":2,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.58599999999999997,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80100000000000005}]},{"osd":3,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.98199999999999998,"5min":1.218,"15min":1.218},"last":1.0209999999999999}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.84399999999999997,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.054,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.89000000000000001}]}]},{"osd":0,"up_from":12,"seq":51539607594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36624,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335216,"statfs":{"total":96636764160,"available":96599261184,"internally_reserved":0,"allocated":1441792,"data_stored":563023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.753},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.87}]},{"osd":2,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":1.0309999999999999}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54800000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.262},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":1.198}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.095}]},{"osd":5,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.81299999999999994,"15min":0.79900000000000004},"min":{"1min":0.60699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":1.151},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.163,"5min":1.694,"15min":1.694},"last":1.0609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:21:17.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:21:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:17.637 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:21:17.653 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-13T05:21:16.096016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8392,"num_read_kb":199260,"num_write":1143,"num_write_kb":275373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":162198480,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1243,"ondisk_log_size":1243,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":202520,"kb_used_data":9532,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":566028520,"statfs":{"total":579820584960,"available":579613204480,"internally_reserved":0,"allocated":9760768,"data_stored":4535618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.935329"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945720+0000","last_change":"2024-09-13T05:21:09.054560+0000","last_active":"2024-09-13T05:21:13.945720+0000","last_peered":"2024-09-13T05:21:13.945720+0000","last_clean":"2024-09-13T05:21:13.945720+0000","last_became_active":"2024-09-13T05:21:09.054143+0000","last_became_peered":"2024-09-13T05:21:09.054143+0000","last_unstale":"2024-09-13T05:21:13.945720+0000","last_undegraded":"2024-09-13T05:21:13.945720+0000","last_fullsized":"2024-09-13T05:21:13.945720+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:53.562792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"107'398","reported_seq":2741,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945778+0000","last_change":"2024-09-13T05:21:09.054383+0000","last_active":"2024-09-13T05:21:13.945778+0000","last_peered":"2024-09-13T05:21:13.945778+0000","last_clean":"2024-09-13T05:21:13.945778+0000","last_became_active":"2024-09-13T05:21:09.053687+0000","last_became_peered":"2024-09-13T05:21:09.053687+0000","last_unstale":"2024-09-13T05:21:13.945778+0000","last_undegraded":"2024-09-13T05:21:13.945778+0000","last_fullsized":"2024-09-13T05:21:13.945778+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:41.805028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070414999999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5324,"num_read_kb":28521,"num_write":392,"num_write_kb":20803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":12648486,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"107'91","reported_seq":436,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.180355+0000","last_change":"2024-09-13T05:21:00.105385+0000","last_active":"2024-09-13T05:21:13.180355+0000","last_peered":"2024-09-13T05:21:13.180355+0000","last_clean":"2024-09-13T05:21:13.180355+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:21:13.180355+0000","last_undegraded":"2024-09-13T05:21:13.180355+0000","last_fullsized":"2024-09-13T05:21:13.180355+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074870500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":27203,"num_write":68,"num_write_kb":49155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"107'36","reported_seq":111,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.181366+0000","last_change":"2024-09-13T05:21:00.107196+0000","last_active":"2024-09-13T05:21:13.181366+0000","last_peered":"2024-09-13T05:21:13.181366+0000","last_clean":"2024-09-13T05:21:13.181366+0000","last_became_active":"2024-09-13T05:20:36.954764+0000","last_became_peered":"2024-09-13T05:20:36.954764+0000","last_unstale":"2024-09-13T05:21:13.181366+0000","last_undegraded":"2024-09-13T05:21:13.181366+0000","last_fullsized":"2024-09-13T05:21:13.181366+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:11:18.106127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054390799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":8215,"num_write":27,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"106'104","reported_seq":565,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.935859+0000","last_change":"2024-09-13T05:21:00.114748+0000","last_active":"2024-09-13T05:21:13.935859+0000","last_peered":"2024-09-13T05:21:13.935859+0000","last_clean":"2024-09-13T05:21:13.935859+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:21:13.935859+0000","last_undegraded":"2024-09-13T05:21:13.935859+0000","last_fullsized":"2024-09-13T05:21:13.935859+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070499600000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":38450,"num_write":79,"num_write_kb":38939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"107'97","reported_seq":403,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.856968+0000","last_change":"2024-09-13T05:21:00.112104+0000","last_active":"2024-09-13T05:21:12.856968+0000","last_peered":"2024-09-13T05:21:12.856968+0000","last_clean":"2024-09-13T05:21:12.856968+0000","last_became_active":"2024-09-13T05:20:36.957689+0000","last_became_peered":"2024-09-13T05:20:36.957689+0000","last_unstale":"2024-09-13T05:21:12.856968+0000","last_undegraded":"2024-09-13T05:21:12.856968+0000","last_fullsized":"2024-09-13T05:21:12.856968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T06:29:41.023217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001858878,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":16522,"num_write":101,"num_write_kb":32810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":33554508,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"107'73","reported_seq":43,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945775+0000","last_change":"2024-09-13T05:21:06.170554+0000","last_active":"2024-09-13T05:21:13.945775+0000","last_peered":"2024-09-13T05:21:13.945775+0000","last_clean":"2024-09-13T05:21:13.945775+0000","last_became_active":"2024-09-13T05:21:06.157892+0000","last_became_peered":"2024-09-13T05:21:06.157892+0000","last_unstale":"2024-09-13T05:21:13.945775+0000","last_undegraded":"2024-09-13T05:21:13.945775+0000","last_fullsized":"2024-09-13T05:21:13.945775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:52.415500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081846100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":19146,"num_write":56,"num_write_kb":41013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":41992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'84","reported_seq":211,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.857059+0000","last_change":"2024-09-13T05:21:00.112146+0000","last_active":"2024-09-13T05:21:12.857059+0000","last_peered":"2024-09-13T05:21:12.857059+0000","last_clean":"2024-09-13T05:21:12.857059+0000","last_became_active":"2024-09-13T05:20:43.986894+0000","last_became_peered":"2024-09-13T05:20:43.986894+0000","last_unstale":"2024-09-13T05:21:12.857059+0000","last_undegraded":"2024-09-13T05:21:12.857059+0000","last_fullsized":"2024-09-13T05:21:12.857059+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:15:48.175749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019911320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":516,"num_read_kb":12708,"num_write":57,"num_write_kb":24588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29360244,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"107'243","reported_seq":1609,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945824+0000","last_change":"2024-09-13T05:21:09.052792+0000","last_active":"2024-09-13T05:21:13.945824+0000","last_peered":"2024-09-13T05:21:13.945824+0000","last_clean":"2024-09-13T05:21:13.945824+0000","last_became_active":"2024-09-13T05:21:09.051755+0000","last_became_peered":"2024-09-13T05:21:09.051755+0000","last_unstale":"2024-09-13T05:21:13.945824+0000","last_undegraded":"2024-09-13T05:21:13.945824+0000","last_fullsized":"2024-09-13T05:21:13.945824+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:32.347051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291826,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1348,"num_read_kb":48395,"num_write":204,"num_write_kb":49262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":23080975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8276,"num_read_kb":199160,"num_write":984,"num_write_kb":272957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":161608112,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1126,"ondisk_log_size":1126,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607595,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":2007040,"data_stored":1144100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.071,"15min":1.071},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.89700000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51900000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.028}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0580000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Sep 13 05:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.997}]}]},{"osd":4,"up_from":54,"seq":231928234006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29488,"kb_used_data":1380,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342352,"statfs":{"total":96636764160,"available":96606568448,"internally_reserved":0,"allocated":1413120,"data_stored":553732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.95199999999999996}]},{"osd":1,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.83899999999999997}]},{"osd":2,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.78700000000000003}]},{"osd":3,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.86499999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.753},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.92900000000000005}]}]},{"osd":3,"up_from":29,"seq":124554051616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":1.246},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69599999999999995,"15min":0.69999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":1.222}]},{"osd":1,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68300000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.627}]},{"osd":2,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67800000000000005,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92000000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.53200000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.96699999999999997}]},{"osd":4,"last update":"Fri Sep 13 05:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.1779999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68300000000000005,"15min":0.66800000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.105},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.89400000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36608,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1425408,"data_stored":558368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.28299999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.95899999999999996,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.59099999999999997},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.083,"5min":1.171,"15min":1.171},"last":1.083},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86099999999999999,"5min":1.21,"15min":1.21},"last":0.755}]},{"osd":3,"last update":"Fri Sep 13 05:20:45 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.90900000000000003,"5min":1.083,"15min":1.083},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.751,"15min":0.752},"min":{"1min":0.54700000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93700000000000006,"5min":1.21,"15min":1.21},"last":1.028}]},{"osd":4,"last update":"Fri Sep 13 05:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":1.206}]},{"osd":5,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3799999999999999,"5min":1.544,"15min":1.544},"last":1.3089999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.1399999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607594,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":2035712,"data_stored":1153391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88800000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":1.119}]},{"osd":2,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.58599999999999997,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80100000000000005}]},{"osd":3,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.98199999999999998,"5min":1.218,"15min":1.218},"last":1.0209999999999999}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.84399999999999997,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.054,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.89000000000000001}]}]},{"osd":0,"up_from":12,"seq":51539607594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36624,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335216,"statfs":{"total":96636764160,"available":96599261184,"internally_reserved":0,"allocated":1441792,"data_stored":563023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.753},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.87}]},{"osd":2,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":1.0309999999999999}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54800000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.262},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":1.198}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.095}]},{"osd":5,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.81299999999999994,"15min":0.79900000000000004},"min":{"1min":0.60699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":1.151},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.163,"5min":1.694,"15min":1.694},"last":1.0609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:21:17.654 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T05:21:17.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-13T05:21:17.970 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T05:21:17.979 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T05:21:17.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:21:18.273 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:18.273 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:21:18.290 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-13T05:21:18.096601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8392,"num_read_kb":199260,"num_write":1143,"num_write_kb":275373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":162198480,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1243,"ondisk_log_size":1243,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":202520,"kb_used_data":9532,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":566028520,"statfs":{"total":579820584960,"available":579613204480,"internally_reserved":0,"allocated":9760768,"data_stored":4535618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001693"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945720+0000","last_change":"2024-09-13T05:21:09.054560+0000","last_active":"2024-09-13T05:21:13.945720+0000","last_peered":"2024-09-13T05:21:13.945720+0000","last_clean":"2024-09-13T05:21:13.945720+0000","last_became_active":"2024-09-13T05:21:09.054143+0000","last_became_peered":"2024-09-13T05:21:09.054143+0000","last_unstale":"2024-09-13T05:21:13.945720+0000","last_undegraded":"2024-09-13T05:21:13.945720+0000","last_fullsized":"2024-09-13T05:21:13.945720+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:53.562792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"107'398","reported_seq":2741,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945778+0000","last_change":"2024-09-13T05:21:09.054383+0000","last_active":"2024-09-13T05:21:13.945778+0000","last_peered":"2024-09-13T05:21:13.945778+0000","last_clean":"2024-09-13T05:21:13.945778+0000","last_became_active":"2024-09-13T05:21:09.053687+0000","last_became_peered":"2024-09-13T05:21:09.053687+0000","last_unstale":"2024-09-13T05:21:13.945778+0000","last_undegraded":"2024-09-13T05:21:13.945778+0000","last_fullsized":"2024-09-13T05:21:13.945778+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:41.805028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070414999999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5324,"num_read_kb":28521,"num_write":392,"num_write_kb":20803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":12648486,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"107'91","reported_seq":436,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.180355+0000","last_change":"2024-09-13T05:21:00.105385+0000","last_active":"2024-09-13T05:21:13.180355+0000","last_peered":"2024-09-13T05:21:13.180355+0000","last_clean":"2024-09-13T05:21:13.180355+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:21:13.180355+0000","last_undegraded":"2024-09-13T05:21:13.180355+0000","last_fullsized":"2024-09-13T05:21:13.180355+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074870500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":27203,"num_write":68,"num_write_kb":49155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"107'36","reported_seq":111,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.181366+0000","last_change":"2024-09-13T05:21:00.107196+0000","last_active":"2024-09-13T05:21:13.181366+0000","last_peered":"2024-09-13T05:21:13.181366+0000","last_clean":"2024-09-13T05:21:13.181366+0000","last_became_active":"2024-09-13T05:20:36.954764+0000","last_became_peered":"2024-09-13T05:20:36.954764+0000","last_unstale":"2024-09-13T05:21:13.181366+0000","last_undegraded":"2024-09-13T05:21:13.181366+0000","last_fullsized":"2024-09-13T05:21:13.181366+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:11:18.106127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054390799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":8215,"num_write":27,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"106'104","reported_seq":565,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.935859+0000","last_change":"2024-09-13T05:21:00.114748+0000","last_active":"2024-09-13T05:21:13.935859+0000","last_peered":"2024-09-13T05:21:13.935859+0000","last_clean":"2024-09-13T05:21:13.935859+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:21:13.935859+0000","last_undegraded":"2024-09-13T05:21:13.935859+0000","last_fullsized":"2024-09-13T05:21:13.935859+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070499600000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":38450,"num_write":79,"num_write_kb":38939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"107'97","reported_seq":403,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.856968+0000","last_change":"2024-09-13T05:21:00.112104+0000","last_active":"2024-09-13T05:21:12.856968+0000","last_peered":"2024-09-13T05:21:12.856968+0000","last_clean":"2024-09-13T05:21:12.856968+0000","last_became_active":"2024-09-13T05:20:36.957689+0000","last_became_peered":"2024-09-13T05:20:36.957689+0000","last_unstale":"2024-09-13T05:21:12.856968+0000","last_undegraded":"2024-09-13T05:21:12.856968+0000","last_fullsized":"2024-09-13T05:21:12.856968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T06:29:41.023217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001858878,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":16522,"num_write":101,"num_write_kb":32810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":33554508,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"107'73","reported_seq":43,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945775+0000","last_change":"2024-09-13T05:21:06.170554+0000","last_active":"2024-09-13T05:21:13.945775+0000","last_peered":"2024-09-13T05:21:13.945775+0000","last_clean":"2024-09-13T05:21:13.945775+0000","last_became_active":"2024-09-13T05:21:06.157892+0000","last_became_peered":"2024-09-13T05:21:06.157892+0000","last_unstale":"2024-09-13T05:21:13.945775+0000","last_undegraded":"2024-09-13T05:21:13.945775+0000","last_fullsized":"2024-09-13T05:21:13.945775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:52.415500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081846100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":19146,"num_write":56,"num_write_kb":41013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":41992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'84","reported_seq":211,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.857059+0000","last_change":"2024-09-13T05:21:00.112146+0000","last_active":"2024-09-13T05:21:12.857059+0000","last_peered":"2024-09-13T05:21:12.857059+0000","last_clean":"2024-09-13T05:21:12.857059+0000","last_became_active":"2024-09-13T05:20:43.986894+0000","last_became_peered":"2024-09-13T05:20:43.986894+0000","last_unstale":"2024-09-13T05:21:12.857059+0000","last_undegraded":"2024-09-13T05:21:12.857059+0000","last_fullsized":"2024-09-13T05:21:12.857059+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:15:48.175749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019911320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":516,"num_read_kb":12708,"num_write":57,"num_write_kb":24588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29360244,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"107'243","reported_seq":1609,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945824+0000","last_change":"2024-09-13T05:21:09.052792+0000","last_active":"2024-09-13T05:21:13.945824+0000","last_peered":"2024-09-13T05:21:13.945824+0000","last_clean":"2024-09-13T05:21:13.945824+0000","last_became_active":"2024-09-13T05:21:09.051755+0000","last_became_peered":"2024-09-13T05:21:09.051755+0000","last_unstale":"2024-09-13T05:21:13.945824+0000","last_undegraded":"2024-09-13T05:21:13.945824+0000","last_fullsized":"2024-09-13T05:21:13.945824+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:32.347051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291826,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1348,"num_read_kb":48395,"num_write":204,"num_write_kb":49262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":23080975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8276,"num_read_kb":199160,"num_write":984,"num_write_kb":272957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":161608112,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1126,"ondisk_log_size":1126,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607595,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":2007040,"data_stored":1144100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.071,"15min":1.071},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.89700000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51900000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.028}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0580000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Sep 13 05:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.997}]}]},{"osd":4,"up_from":54,"seq":231928234006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29488,"kb_used_data":1380,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342352,"statfs":{"total":96636764160,"available":96606568448,"internally_reserved":0,"allocated":1413120,"data_stored":553732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.95199999999999996}]},{"osd":1,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.83899999999999997}]},{"osd":2,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.78700000000000003}]},{"osd":3,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.86499999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.753},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.92900000000000005}]}]},{"osd":3,"up_from":29,"seq":124554051617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":1.246},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69599999999999995,"15min":0.69999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":1.222}]},{"osd":1,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68300000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.627}]},{"osd":2,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67800000000000005,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92000000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.53200000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.96699999999999997}]},{"osd":4,"last update":"Fri Sep 13 05:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.1779999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68300000000000005,"15min":0.66800000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.105},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.89400000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36608,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1425408,"data_stored":558368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.28299999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.95899999999999996,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.59099999999999997},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.083,"5min":1.171,"15min":1.171},"last":1.083},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86099999999999999,"5min":1.21,"15min":1.21},"last":0.755}]},{"osd":3,"last update":"Fri Sep 13 05:20:45 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.90900000000000003,"5min":1.083,"15min":1.083},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.751,"15min":0.752},"min":{"1min":0.54700000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93700000000000006,"5min":1.21,"15min":1.21},"last":1.028}]},{"osd":4,"last update":"Fri Sep 13 05:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":1.206}]},{"osd":5,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3799999999999999,"5min":1.544,"15min":1.544},"last":1.3089999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.1399999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607594,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":2035712,"data_stored":1153391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88800000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":1.119}]},{"osd":2,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.58599999999999997,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80100000000000005}]},{"osd":3,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.98199999999999998,"5min":1.218,"15min":1.218},"last":1.0209999999999999}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.84399999999999997,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.054,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.89000000000000001}]}]},{"osd":0,"up_from":12,"seq":51539607594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36624,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335216,"statfs":{"total":96636764160,"available":96599261184,"internally_reserved":0,"allocated":1441792,"data_stored":563023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.753},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.87}]},{"osd":2,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":1.0309999999999999}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54800000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.262},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":1.198}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.095}]},{"osd":5,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.81299999999999994,"15min":0.79900000000000004},"min":{"1min":0.60699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":1.151},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.163,"5min":1.694,"15min":1.694},"last":1.0609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:21:18.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:21:18.578 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:18.578 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:21:18.594 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-13T05:21:18.096601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8392,"num_read_kb":199260,"num_write":1143,"num_write_kb":275373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":162198480,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1243,"ondisk_log_size":1243,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":202520,"kb_used_data":9532,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":566028520,"statfs":{"total":579820584960,"available":579613204480,"internally_reserved":0,"allocated":9760768,"data_stored":4535618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001693"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945720+0000","last_change":"2024-09-13T05:21:09.054560+0000","last_active":"2024-09-13T05:21:13.945720+0000","last_peered":"2024-09-13T05:21:13.945720+0000","last_clean":"2024-09-13T05:21:13.945720+0000","last_became_active":"2024-09-13T05:21:09.054143+0000","last_became_peered":"2024-09-13T05:21:09.054143+0000","last_unstale":"2024-09-13T05:21:13.945720+0000","last_undegraded":"2024-09-13T05:21:13.945720+0000","last_fullsized":"2024-09-13T05:21:13.945720+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:53.562792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"107'398","reported_seq":2741,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945778+0000","last_change":"2024-09-13T05:21:09.054383+0000","last_active":"2024-09-13T05:21:13.945778+0000","last_peered":"2024-09-13T05:21:13.945778+0000","last_clean":"2024-09-13T05:21:13.945778+0000","last_became_active":"2024-09-13T05:21:09.053687+0000","last_became_peered":"2024-09-13T05:21:09.053687+0000","last_unstale":"2024-09-13T05:21:13.945778+0000","last_undegraded":"2024-09-13T05:21:13.945778+0000","last_fullsized":"2024-09-13T05:21:13.945778+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:41.805028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070414999999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5324,"num_read_kb":28521,"num_write":392,"num_write_kb":20803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":12648486,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"107'91","reported_seq":436,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.180355+0000","last_change":"2024-09-13T05:21:00.105385+0000","last_active":"2024-09-13T05:21:13.180355+0000","last_peered":"2024-09-13T05:21:13.180355+0000","last_clean":"2024-09-13T05:21:13.180355+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:21:13.180355+0000","last_undegraded":"2024-09-13T05:21:13.180355+0000","last_fullsized":"2024-09-13T05:21:13.180355+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074870500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":27203,"num_write":68,"num_write_kb":49155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"107'36","reported_seq":111,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.181366+0000","last_change":"2024-09-13T05:21:00.107196+0000","last_active":"2024-09-13T05:21:13.181366+0000","last_peered":"2024-09-13T05:21:13.181366+0000","last_clean":"2024-09-13T05:21:13.181366+0000","last_became_active":"2024-09-13T05:20:36.954764+0000","last_became_peered":"2024-09-13T05:20:36.954764+0000","last_unstale":"2024-09-13T05:21:13.181366+0000","last_undegraded":"2024-09-13T05:21:13.181366+0000","last_fullsized":"2024-09-13T05:21:13.181366+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:11:18.106127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054390799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":8215,"num_write":27,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"106'104","reported_seq":565,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.935859+0000","last_change":"2024-09-13T05:21:00.114748+0000","last_active":"2024-09-13T05:21:13.935859+0000","last_peered":"2024-09-13T05:21:13.935859+0000","last_clean":"2024-09-13T05:21:13.935859+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:21:13.935859+0000","last_undegraded":"2024-09-13T05:21:13.935859+0000","last_fullsized":"2024-09-13T05:21:13.935859+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070499600000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":38450,"num_write":79,"num_write_kb":38939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"107'97","reported_seq":403,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.856968+0000","last_change":"2024-09-13T05:21:00.112104+0000","last_active":"2024-09-13T05:21:12.856968+0000","last_peered":"2024-09-13T05:21:12.856968+0000","last_clean":"2024-09-13T05:21:12.856968+0000","last_became_active":"2024-09-13T05:20:36.957689+0000","last_became_peered":"2024-09-13T05:20:36.957689+0000","last_unstale":"2024-09-13T05:21:12.856968+0000","last_undegraded":"2024-09-13T05:21:12.856968+0000","last_fullsized":"2024-09-13T05:21:12.856968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T06:29:41.023217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001858878,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":16522,"num_write":101,"num_write_kb":32810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":33554508,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"107'73","reported_seq":43,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945775+0000","last_change":"2024-09-13T05:21:06.170554+0000","last_active":"2024-09-13T05:21:13.945775+0000","last_peered":"2024-09-13T05:21:13.945775+0000","last_clean":"2024-09-13T05:21:13.945775+0000","last_became_active":"2024-09-13T05:21:06.157892+0000","last_became_peered":"2024-09-13T05:21:06.157892+0000","last_unstale":"2024-09-13T05:21:13.945775+0000","last_undegraded":"2024-09-13T05:21:13.945775+0000","last_fullsized":"2024-09-13T05:21:13.945775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:52.415500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081846100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":19146,"num_write":56,"num_write_kb":41013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":41992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'84","reported_seq":211,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.857059+0000","last_change":"2024-09-13T05:21:00.112146+0000","last_active":"2024-09-13T05:21:12.857059+0000","last_peered":"2024-09-13T05:21:12.857059+0000","last_clean":"2024-09-13T05:21:12.857059+0000","last_became_active":"2024-09-13T05:20:43.986894+0000","last_became_peered":"2024-09-13T05:20:43.986894+0000","last_unstale":"2024-09-13T05:21:12.857059+0000","last_undegraded":"2024-09-13T05:21:12.857059+0000","last_fullsized":"2024-09-13T05:21:12.857059+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:15:48.175749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019911320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":516,"num_read_kb":12708,"num_write":57,"num_write_kb":24588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29360244,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"107'243","reported_seq":1609,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945824+0000","last_change":"2024-09-13T05:21:09.052792+0000","last_active":"2024-09-13T05:21:13.945824+0000","last_peered":"2024-09-13T05:21:13.945824+0000","last_clean":"2024-09-13T05:21:13.945824+0000","last_became_active":"2024-09-13T05:21:09.051755+0000","last_became_peered":"2024-09-13T05:21:09.051755+0000","last_unstale":"2024-09-13T05:21:13.945824+0000","last_undegraded":"2024-09-13T05:21:13.945824+0000","last_fullsized":"2024-09-13T05:21:13.945824+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:32.347051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291826,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1348,"num_read_kb":48395,"num_write":204,"num_write_kb":49262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":23080975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8276,"num_read_kb":199160,"num_write":984,"num_write_kb":272957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":161608112,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1126,"ondisk_log_size":1126,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607595,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":2007040,"data_stored":1144100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.071,"15min":1.071},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.89700000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51900000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.028}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0580000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Sep 13 05:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.997}]}]},{"osd":4,"up_from":54,"seq":231928234006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29488,"kb_used_data":1380,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342352,"statfs":{"total":96636764160,"available":96606568448,"internally_reserved":0,"allocated":1413120,"data_stored":553732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.95199999999999996}]},{"osd":1,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.83899999999999997}]},{"osd":2,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.78700000000000003}]},{"osd":3,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.86499999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.753},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.92900000000000005}]}]},{"osd":3,"up_from":29,"seq":124554051617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":1.246},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69599999999999995,"15min":0.69999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":1.222}]},{"osd":1,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68300000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.627}]},{"osd":2,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67800000000000005,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92000000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.53200000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.96699999999999997}]},{"osd":4,"last update":"Fri Sep 13 05:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.1779999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68300000000000005,"15min":0.66800000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.105},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.89400000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36608,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1425408,"data_stored":558368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.28299999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.95899999999999996,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.59099999999999997},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.083,"5min":1.171,"15min":1.171},"last":1.083},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86099999999999999,"5min":1.21,"15min":1.21},"last":0.755}]},{"osd":3,"last update":"Fri Sep 13 05:20:45 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.90900000000000003,"5min":1.083,"15min":1.083},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.751,"15min":0.752},"min":{"1min":0.54700000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93700000000000006,"5min":1.21,"15min":1.21},"last":1.028}]},{"osd":4,"last update":"Fri Sep 13 05:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":1.206}]},{"osd":5,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3799999999999999,"5min":1.544,"15min":1.544},"last":1.3089999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.1399999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607594,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":2035712,"data_stored":1153391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88800000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":1.119}]},{"osd":2,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.58599999999999997,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80100000000000005}]},{"osd":3,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.98199999999999998,"5min":1.218,"15min":1.218},"last":1.0209999999999999}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.84399999999999997,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.054,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.89000000000000001}]}]},{"osd":0,"up_from":12,"seq":51539607594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36624,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335216,"statfs":{"total":96636764160,"available":96599261184,"internally_reserved":0,"allocated":1441792,"data_stored":563023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.753},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.87}]},{"osd":2,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":1.0309999999999999}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54800000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.262},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":1.198}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.095}]},{"osd":5,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.81299999999999994,"15min":0.79900000000000004},"min":{"1min":0.60699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":1.151},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.163,"5min":1.694,"15min":1.694},"last":1.0609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:21:18.596 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T05:21:18.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:21:18.883 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:18.884 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:21:18.903 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-13T05:21:18.096601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8392,"num_read_kb":199260,"num_write":1143,"num_write_kb":275373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":162198480,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1243,"ondisk_log_size":1243,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":202520,"kb_used_data":9532,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":566028520,"statfs":{"total":579820584960,"available":579613204480,"internally_reserved":0,"allocated":9760768,"data_stored":4535618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001693"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945720+0000","last_change":"2024-09-13T05:21:09.054560+0000","last_active":"2024-09-13T05:21:13.945720+0000","last_peered":"2024-09-13T05:21:13.945720+0000","last_clean":"2024-09-13T05:21:13.945720+0000","last_became_active":"2024-09-13T05:21:09.054143+0000","last_became_peered":"2024-09-13T05:21:09.054143+0000","last_unstale":"2024-09-13T05:21:13.945720+0000","last_undegraded":"2024-09-13T05:21:13.945720+0000","last_fullsized":"2024-09-13T05:21:13.945720+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:53.562792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"107'398","reported_seq":2741,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945778+0000","last_change":"2024-09-13T05:21:09.054383+0000","last_active":"2024-09-13T05:21:13.945778+0000","last_peered":"2024-09-13T05:21:13.945778+0000","last_clean":"2024-09-13T05:21:13.945778+0000","last_became_active":"2024-09-13T05:21:09.053687+0000","last_became_peered":"2024-09-13T05:21:09.053687+0000","last_unstale":"2024-09-13T05:21:13.945778+0000","last_undegraded":"2024-09-13T05:21:13.945778+0000","last_fullsized":"2024-09-13T05:21:13.945778+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:41.805028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070414999999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5324,"num_read_kb":28521,"num_write":392,"num_write_kb":20803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":12648486,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"107'91","reported_seq":436,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.180355+0000","last_change":"2024-09-13T05:21:00.105385+0000","last_active":"2024-09-13T05:21:13.180355+0000","last_peered":"2024-09-13T05:21:13.180355+0000","last_clean":"2024-09-13T05:21:13.180355+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:21:13.180355+0000","last_undegraded":"2024-09-13T05:21:13.180355+0000","last_fullsized":"2024-09-13T05:21:13.180355+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074870500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":27203,"num_write":68,"num_write_kb":49155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"107'36","reported_seq":111,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.181366+0000","last_change":"2024-09-13T05:21:00.107196+0000","last_active":"2024-09-13T05:21:13.181366+0000","last_peered":"2024-09-13T05:21:13.181366+0000","last_clean":"2024-09-13T05:21:13.181366+0000","last_became_active":"2024-09-13T05:20:36.954764+0000","last_became_peered":"2024-09-13T05:20:36.954764+0000","last_unstale":"2024-09-13T05:21:13.181366+0000","last_undegraded":"2024-09-13T05:21:13.181366+0000","last_fullsized":"2024-09-13T05:21:13.181366+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:11:18.106127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054390799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":8215,"num_write":27,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"106'104","reported_seq":565,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.935859+0000","last_change":"2024-09-13T05:21:00.114748+0000","last_active":"2024-09-13T05:21:13.935859+0000","last_peered":"2024-09-13T05:21:13.935859+0000","last_clean":"2024-09-13T05:21:13.935859+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:21:13.935859+0000","last_undegraded":"2024-09-13T05:21:13.935859+0000","last_fullsized":"2024-09-13T05:21:13.935859+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070499600000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":38450,"num_write":79,"num_write_kb":38939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"107'97","reported_seq":403,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.856968+0000","last_change":"2024-09-13T05:21:00.112104+0000","last_active":"2024-09-13T05:21:12.856968+0000","last_peered":"2024-09-13T05:21:12.856968+0000","last_clean":"2024-09-13T05:21:12.856968+0000","last_became_active":"2024-09-13T05:20:36.957689+0000","last_became_peered":"2024-09-13T05:20:36.957689+0000","last_unstale":"2024-09-13T05:21:12.856968+0000","last_undegraded":"2024-09-13T05:21:12.856968+0000","last_fullsized":"2024-09-13T05:21:12.856968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T06:29:41.023217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001858878,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":16522,"num_write":101,"num_write_kb":32810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":33554508,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"107'73","reported_seq":43,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945775+0000","last_change":"2024-09-13T05:21:06.170554+0000","last_active":"2024-09-13T05:21:13.945775+0000","last_peered":"2024-09-13T05:21:13.945775+0000","last_clean":"2024-09-13T05:21:13.945775+0000","last_became_active":"2024-09-13T05:21:06.157892+0000","last_became_peered":"2024-09-13T05:21:06.157892+0000","last_unstale":"2024-09-13T05:21:13.945775+0000","last_undegraded":"2024-09-13T05:21:13.945775+0000","last_fullsized":"2024-09-13T05:21:13.945775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:52.415500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081846100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":19146,"num_write":56,"num_write_kb":41013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":41992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'84","reported_seq":211,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.857059+0000","last_change":"2024-09-13T05:21:00.112146+0000","last_active":"2024-09-13T05:21:12.857059+0000","last_peered":"2024-09-13T05:21:12.857059+0000","last_clean":"2024-09-13T05:21:12.857059+0000","last_became_active":"2024-09-13T05:20:43.986894+0000","last_became_peered":"2024-09-13T05:20:43.986894+0000","last_unstale":"2024-09-13T05:21:12.857059+0000","last_undegraded":"2024-09-13T05:21:12.857059+0000","last_fullsized":"2024-09-13T05:21:12.857059+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:15:48.175749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019911320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":516,"num_read_kb":12708,"num_write":57,"num_write_kb":24588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29360244,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"107'243","reported_seq":1609,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945824+0000","last_change":"2024-09-13T05:21:09.052792+0000","last_active":"2024-09-13T05:21:13.945824+0000","last_peered":"2024-09-13T05:21:13.945824+0000","last_clean":"2024-09-13T05:21:13.945824+0000","last_became_active":"2024-09-13T05:21:09.051755+0000","last_became_peered":"2024-09-13T05:21:09.051755+0000","last_unstale":"2024-09-13T05:21:13.945824+0000","last_undegraded":"2024-09-13T05:21:13.945824+0000","last_fullsized":"2024-09-13T05:21:13.945824+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:32.347051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291826,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1348,"num_read_kb":48395,"num_write":204,"num_write_kb":49262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":23080975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8276,"num_read_kb":199160,"num_write":984,"num_write_kb":272957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":161608112,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1126,"ondisk_log_size":1126,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607595,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":2007040,"data_stored":1144100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.071,"15min":1.071},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.89700000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51900000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.028}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0580000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Sep 13 05:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.997}]}]},{"osd":4,"up_from":54,"seq":231928234006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29488,"kb_used_data":1380,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342352,"statfs":{"total":96636764160,"available":96606568448,"internally_reserved":0,"allocated":1413120,"data_stored":553732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.95199999999999996}]},{"osd":1,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.83899999999999997}]},{"osd":2,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.78700000000000003}]},{"osd":3,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.86499999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.753},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.92900000000000005}]}]},{"osd":3,"up_from":29,"seq":124554051617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":1.246},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69599999999999995,"15min":0.69999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":1.222}]},{"osd":1,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68300000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.627}]},{"osd":2,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67800000000000005,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92000000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.53200000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.96699999999999997}]},{"osd":4,"last update":"Fri Sep 13 05:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.1779999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68300000000000005,"15min":0.66800000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.105},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.89400000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36608,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1425408,"data_stored":558368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.28299999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.95899999999999996,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.59099999999999997},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.083,"5min":1.171,"15min":1.171},"last":1.083},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86099999999999999,"5min":1.21,"15min":1.21},"last":0.755}]},{"osd":3,"last update":"Fri Sep 13 05:20:45 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.90900000000000003,"5min":1.083,"15min":1.083},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.751,"15min":0.752},"min":{"1min":0.54700000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93700000000000006,"5min":1.21,"15min":1.21},"last":1.028}]},{"osd":4,"last update":"Fri Sep 13 05:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":1.206}]},{"osd":5,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3799999999999999,"5min":1.544,"15min":1.544},"last":1.3089999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.1399999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607594,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":2035712,"data_stored":1153391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88800000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":1.119}]},{"osd":2,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.58599999999999997,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68200000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80100000000000005}]},{"osd":3,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.98199999999999998,"5min":1.218,"15min":1.218},"last":1.0209999999999999}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.84399999999999997,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.054,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.89000000000000001}]}]},{"osd":0,"up_from":12,"seq":51539607594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36624,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335216,"statfs":{"total":96636764160,"available":96599261184,"internally_reserved":0,"allocated":1441792,"data_stored":563023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.753},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.87}]},{"osd":2,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":1.0309999999999999}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54800000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.262},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":1.198}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.095}]},{"osd":5,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.81299999999999994,"15min":0.79900000000000004},"min":{"1min":0.60699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":1.151},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.163,"5min":1.694,"15min":1.694},"last":1.0609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:21:18.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T05:21:19.203 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:19.203 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":121,"fsid":"1e2dd7d5-3bc8-47f3-ada4-e91cfed8c2f5","created":"2024-09-13T05:17:57.363809+0000","modified":"2024-09-13T05:21:14.182493+0000","last_up_change":"2024-09-13T05:19:34.047702+0000","last_in_change":"2024-09-13T05:20:35.926417+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T05:18:08.122164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T05:18:15.001664+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e58174f4-85ec-4321-8bc3-384261c59c6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6801","nonce":2153409514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6803","nonce":2153409514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6807","nonce":2153409514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2153409514},{"type":"v1","addr":"172.21.15.16:6805","nonce":2153409514}]},"public_addr":"172.21.15.16:6801/2153409514","cluster_addr":"172.21.15.16:6803/2153409514","heartbeat_back_addr":"172.21.15.16:6807/2153409514","heartbeat_front_addr":"172.21.15.16:6805/2153409514","state":["exists","up"]},{"osd":1,"uuid":"84a3db6c-dacd-43d5-b964-37fdef0091a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6809","nonce":2608485151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6811","nonce":2608485151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6815","nonce":2608485151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2608485151},{"type":"v1","addr":"172.21.15.16:6813","nonce":2608485151}]},"public_addr":"172.21.15.16:6809/2608485151","cluster_addr":"172.21.15.16:6811/2608485151","heartbeat_back_addr":"172.21.15.16:6815/2608485151","heartbeat_front_addr":"172.21.15.16:6813/2608485151","state":["exists","up"]},{"osd":2,"uuid":"0d530476-cf38-4544-9ce7-e8ed35df220b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6817","nonce":1006404377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6819","nonce":1006404377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6823","nonce":1006404377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1006404377},{"type":"v1","addr":"172.21.15.16:6821","nonce":1006404377}]},"public_addr":"172.21.15.16:6817/1006404377","cluster_addr":"172.21.15.16:6819/1006404377","heartbeat_back_addr":"172.21.15.16:6823/1006404377","heartbeat_front_addr":"172.21.15.16:6821/1006404377","state":["exists","up"]},{"osd":3,"uuid":"fd91f7e2-580d-47ea-9db5-792c8f1ebe01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":12,"last_clean_end":20,"up_from":29,"up_thru":90,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6817","nonce":1834661236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6819","nonce":1834661236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6823","nonce":1834661236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1834661236},{"type":"v1","addr":"172.21.15.79:6821","nonce":1834661236}]},"public_addr":"172.21.15.79:6817/1834661236","cluster_addr":"172.21.15.79:6819/1834661236","heartbeat_back_addr":"172.21.15.79:6823/1834661236","heartbeat_front_addr":"172.21.15.79:6821/1834661236","state":["exists","up"]},{"osd":4,"uuid":"01af0f4c-5da2-46fe-a262-59fcc6a2efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":12,"last_clean_end":38,"up_from":54,"up_thru":87,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6809","nonce":3457694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6811","nonce":3457694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6815","nonce":3457694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3457694380},{"type":"v1","addr":"172.21.15.79:6813","nonce":3457694380}]},"public_addr":"172.21.15.79:6809/3457694380","cluster_addr":"172.21.15.79:6811/3457694380","heartbeat_back_addr":"172.21.15.79:6815/3457694380","heartbeat_front_addr":"172.21.15.79:6813/3457694380","state":["exists","up"]},{"osd":5,"uuid":"64cdd5f2-1568-4e96-ad90-6ef7414651f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6801","nonce":2778785304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6803","nonce":2778785304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6807","nonce":2778785304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2778785304},{"type":"v1","addr":"172.21.15.79:6805","nonce":2778785304}]},"public_addr":"172.21.15.79:6801/2778785304","cluster_addr":"172.21.15.79:6803/2778785304","heartbeat_back_addr":"172.21.15.79:6807/2778785304","heartbeat_front_addr":"172.21.15.79:6805/2778785304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.193869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.255764+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.286354+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.221310+0000","dead_epoch":21},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.026235+0000","dead_epoch":40},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T05:18:05.010242+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.2","osds":[0,1]}],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T05:21:20.224 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T05:21:20.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T05:21:20.386 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T05:21:20.386 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:20.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T05:21:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T05:21:20.712 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T05:21:20.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T05:21:20.868 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:20.868 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T05:21:20.868 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:20.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T05:21:21.182 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T05:21:21.202 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T05:21:21.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T05:21:21.357 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:21.357 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T05:21:21.357 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:21.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T05:21:21.673 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T05:21:21.693 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T05:21:21.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T05:21:21.851 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:21.851 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T05:21:21.852 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:21.868 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T05:21:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T05:21:22.174 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T05:21:22.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T05:21:22.320 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:22.320 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T05:21:22.320 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:22.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T05:21:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T05:21:22.640 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T05:21:22.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T05:21:22.781 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-13T05:21:22.781 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T05:21:22.781 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-13T05:21:22.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T05:21:23.063 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T05:21:23.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:21:23.369 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:21:23.386 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-13T05:21:22.097721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8392,"num_read_kb":199260,"num_write":1143,"num_write_kb":275373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":162198480,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1243,"ondisk_log_size":1243,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":202584,"kb_used_data":9592,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":566028456,"statfs":{"total":579820584960,"available":579613138944,"internally_reserved":0,"allocated":9822208,"data_stored":4558798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.919258"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945720+0000","last_change":"2024-09-13T05:21:09.054560+0000","last_active":"2024-09-13T05:21:13.945720+0000","last_peered":"2024-09-13T05:21:13.945720+0000","last_clean":"2024-09-13T05:21:13.945720+0000","last_became_active":"2024-09-13T05:21:09.054143+0000","last_became_peered":"2024-09-13T05:21:09.054143+0000","last_unstale":"2024-09-13T05:21:13.945720+0000","last_undegraded":"2024-09-13T05:21:13.945720+0000","last_fullsized":"2024-09-13T05:21:13.945720+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:08.785764+0000","last_clean_scrub_stamp":"2024-09-13T05:18:08.785764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:17:53.562792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"107'398","reported_seq":2741,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945778+0000","last_change":"2024-09-13T05:21:09.054383+0000","last_active":"2024-09-13T05:21:13.945778+0000","last_peered":"2024-09-13T05:21:13.945778+0000","last_clean":"2024-09-13T05:21:13.945778+0000","last_became_active":"2024-09-13T05:21:09.053687+0000","last_became_peered":"2024-09-13T05:21:09.053687+0000","last_unstale":"2024-09-13T05:21:13.945778+0000","last_undegraded":"2024-09-13T05:21:13.945778+0000","last_fullsized":"2024-09-13T05:21:13.945778+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:51:41.805028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070414999999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5324,"num_read_kb":28521,"num_write":392,"num_write_kb":20803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":12648486,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"107'91","reported_seq":436,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.180355+0000","last_change":"2024-09-13T05:21:00.105385+0000","last_active":"2024-09-13T05:21:13.180355+0000","last_peered":"2024-09-13T05:21:13.180355+0000","last_clean":"2024-09-13T05:21:13.180355+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:21:13.180355+0000","last_undegraded":"2024-09-13T05:21:13.180355+0000","last_fullsized":"2024-09-13T05:21:13.180355+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:17.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074870500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":27203,"num_write":68,"num_write_kb":49155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"107'36","reported_seq":111,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.181366+0000","last_change":"2024-09-13T05:21:00.107196+0000","last_active":"2024-09-13T05:21:13.181366+0000","last_peered":"2024-09-13T05:21:13.181366+0000","last_clean":"2024-09-13T05:21:13.181366+0000","last_became_active":"2024-09-13T05:20:36.954764+0000","last_became_peered":"2024-09-13T05:20:36.954764+0000","last_unstale":"2024-09-13T05:21:13.181366+0000","last_undegraded":"2024-09-13T05:21:13.181366+0000","last_fullsized":"2024-09-13T05:21:13.181366+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:11:18.106127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054390799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":8215,"num_write":27,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"106'104","reported_seq":565,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.935859+0000","last_change":"2024-09-13T05:21:00.114748+0000","last_active":"2024-09-13T05:21:13.935859+0000","last_peered":"2024-09-13T05:21:13.935859+0000","last_clean":"2024-09-13T05:21:13.935859+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:21:13.935859+0000","last_undegraded":"2024-09-13T05:21:13.935859+0000","last_fullsized":"2024-09-13T05:21:13.935859+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:05:45.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070499600000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":38450,"num_write":79,"num_write_kb":38939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"107'97","reported_seq":403,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.856968+0000","last_change":"2024-09-13T05:21:00.112104+0000","last_active":"2024-09-13T05:21:12.856968+0000","last_peered":"2024-09-13T05:21:12.856968+0000","last_clean":"2024-09-13T05:21:12.856968+0000","last_became_active":"2024-09-13T05:20:36.957689+0000","last_became_peered":"2024-09-13T05:20:36.957689+0000","last_unstale":"2024-09-13T05:21:12.856968+0000","last_undegraded":"2024-09-13T05:21:12.856968+0000","last_fullsized":"2024-09-13T05:21:12.856968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T06:29:41.023217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001858878,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":16522,"num_write":101,"num_write_kb":32810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":33554508,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"107'73","reported_seq":43,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945775+0000","last_change":"2024-09-13T05:21:06.170554+0000","last_active":"2024-09-13T05:21:13.945775+0000","last_peered":"2024-09-13T05:21:13.945775+0000","last_clean":"2024-09-13T05:21:13.945775+0000","last_became_active":"2024-09-13T05:21:06.157892+0000","last_became_peered":"2024-09-13T05:21:06.157892+0000","last_unstale":"2024-09-13T05:21:13.945775+0000","last_undegraded":"2024-09-13T05:21:13.945775+0000","last_fullsized":"2024-09-13T05:21:13.945775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:52.415500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081846100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":19146,"num_write":56,"num_write_kb":41013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":41992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'84","reported_seq":211,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T05:21:12.857059+0000","last_change":"2024-09-13T05:21:00.112146+0000","last_active":"2024-09-13T05:21:12.857059+0000","last_peered":"2024-09-13T05:21:12.857059+0000","last_clean":"2024-09-13T05:21:12.857059+0000","last_became_active":"2024-09-13T05:20:43.986894+0000","last_became_peered":"2024-09-13T05:20:43.986894+0000","last_unstale":"2024-09-13T05:21:12.857059+0000","last_undegraded":"2024-09-13T05:21:12.857059+0000","last_fullsized":"2024-09-13T05:21:12.857059+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:15:48.175749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019911320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":516,"num_read_kb":12708,"num_write":57,"num_write_kb":24588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29360244,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"107'243","reported_seq":1609,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T05:21:13.945824+0000","last_change":"2024-09-13T05:21:09.052792+0000","last_active":"2024-09-13T05:21:13.945824+0000","last_peered":"2024-09-13T05:21:13.945824+0000","last_clean":"2024-09-13T05:21:13.945824+0000","last_became_active":"2024-09-13T05:21:09.051755+0000","last_became_peered":"2024-09-13T05:21:09.051755+0000","last_unstale":"2024-09-13T05:21:13.945824+0000","last_undegraded":"2024-09-13T05:21:13.945824+0000","last_fullsized":"2024-09-13T05:21:13.945824+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:36:32.347051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291826,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1348,"num_read_kb":48395,"num_write":204,"num_write_kb":49262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":23080975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8276,"num_read_kb":199160,"num_write":984,"num_write_kb":272957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":161608112,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1126,"ondisk_log_size":1126,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607596,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":2031616,"data_stored":1153372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.071,"15min":1.071},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.89700000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51900000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.95199999999999996}]},{"osd":2,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.028}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0580000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Sep 13 05:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.997}]}]},{"osd":4,"up_from":54,"seq":231928234007,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.628},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.94999999999999996}]},{"osd":1,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.999}]},{"osd":2,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.89500000000000002}]},{"osd":3,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.78700000000000003}]},{"osd":5,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.97599999999999998}]}]},{"osd":3,"up_from":29,"seq":124554051618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69599999999999995,"15min":0.69999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.81599999999999995}]},{"osd":1,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68300000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.76900000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67800000000000005,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92000000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.53200000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.56999999999999995}]},{"osd":4,"last update":"Fri Sep 13 05:20:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.62},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.85599999999999998}]},{"osd":5,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68300000000000005,"15min":0.66800000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.65700000000000003}]}]},{"osd":2,"up_from":12,"seq":51539607595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36624,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335216,"statfs":{"total":96636764160,"available":96599261184,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.28299999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.95899999999999996,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.59099999999999997},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.083,"5min":1.171,"15min":1.171},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86099999999999999,"5min":1.21,"15min":1.21},"last":0.81799999999999995}]},{"osd":3,"last update":"Fri Sep 13 05:20:45 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.90900000000000003,"5min":1.083,"15min":1.083},"last":0.871},{"interface":"front","average":{"1min":0.746,"5min":0.751,"15min":0.752},"min":{"1min":0.54700000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93700000000000006,"5min":1.21,"15min":1.21},"last":1.004}]},{"osd":4,"last update":"Fri Sep 13 05:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.752},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3799999999999999,"5min":1.544,"15min":1.544},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.0609999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607595,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":2035712,"data_stored":1153391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.96999999999999997,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.625,"15min":0.627},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.42499999999999999}]},{"osd":2,"last update":"Fri Sep 13 05:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56799999999999995,"15min":0.60099999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58699999999999997,"15min":0.61499999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.35799999999999998}]},{"osd":3,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.98199999999999998,"5min":1.218,"15min":1.218},"last":0.626}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Sep 13 05:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.82699999999999996,"15min":0.85299999999999998},"min":{"1min":0.46600000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1359999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.81999999999999995,"15min":0.85599999999999998},"min":{"1min":0.59499999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.056,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.64700000000000002}]}]},{"osd":0,"up_from":12,"seq":51539607595,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36624,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335216,"statfs":{"total":96636764160,"available":96599261184,"internally_reserved":0,"allocated":1441792,"data_stored":563023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.33100000000000002}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54800000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.81299999999999994,"15min":0.79900000000000004},"min":{"1min":0.60699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.163,"5min":1.694,"15min":1.694},"last":0.66100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:21:23.390 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T05:18:08.785764+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:23.391 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T05:18:15.101421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=15, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:23.391 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T05:18:15.101421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=15, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:23.391 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T05:18:15.101421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=15, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:23.391 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T05:18:15.101421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=15, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:23.391 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-13T05:18:15.101421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=15, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:23.391 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T05:18:15.101421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=15, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:23.391 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T05:18:15.101421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=15, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:23.391 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T05:18:15.101421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=15, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:23.391 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T05:21:43.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:21:43.702 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:21:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:21:43.719 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-13T05:21:42.103793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8392,"num_read_kb":199260,"num_write":1143,"num_write_kb":275373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":162198480,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1247,"ondisk_log_size":1247,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":202544,"kb_used_data":9592,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":566028496,"statfs":{"total":579820584960,"available":579613179904,"internally_reserved":0,"allocated":9822208,"data_stored":4558798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003763"},"pg_stats":[{"pgid":"1.0","version":"121'119","reported_seq":194,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:26.780447+0000","last_change":"2024-09-13T05:21:26.780696+0000","last_active":"2024-09-13T05:21:26.780447+0000","last_peered":"2024-09-13T05:21:26.780447+0000","last_clean":"2024-09-13T05:21:26.780447+0000","last_became_active":"2024-09-13T05:21:09.054143+0000","last_became_peered":"2024-09-13T05:21:09.054143+0000","last_unstale":"2024-09-13T05:21:26.780447+0000","last_undegraded":"2024-09-13T05:21:26.780447+0000","last_fullsized":"2024-09-13T05:21:26.780447+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'119","last_scrub_stamp":"2024-09-13T05:21:26.780387+0000","last_deep_scrub":"121'119","last_deep_scrub_stamp":"2024-09-13T05:21:26.780387+0000","last_clean_scrub_stamp":"2024-09-13T05:21:26.780387+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:27:43.835450+0000","scrub_duration":0.0095001479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"121'399","reported_seq":2754,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:35.836118+0000","last_change":"2024-09-13T05:21:35.836396+0000","last_active":"2024-09-13T05:21:35.836118+0000","last_peered":"2024-09-13T05:21:35.836118+0000","last_clean":"2024-09-13T05:21:35.836118+0000","last_became_active":"2024-09-13T05:21:09.053687+0000","last_became_peered":"2024-09-13T05:21:09.053687+0000","last_unstale":"2024-09-13T05:21:35.836118+0000","last_undegraded":"2024-09-13T05:21:35.836118+0000","last_fullsized":"2024-09-13T05:21:35.836118+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'399","last_scrub_stamp":"2024-09-13T05:21:35.836062+0000","last_deep_scrub":"121'399","last_deep_scrub_stamp":"2024-09-13T05:21:35.836062+0000","last_clean_scrub_stamp":"2024-09-13T05:21:35.836062+0000","objects_scrubbed":1,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T05:32:49.408912+0000","scrub_duration":0.0064857179999999997,"objects_trimmed":0,"snaptrim_duration":0.00070414999999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5324,"num_read_kb":28521,"num_write":392,"num_write_kb":20803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":12648486,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"107'91","reported_seq":448,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:23.371088+0000","last_change":"2024-09-13T05:21:23.371366+0000","last_active":"2024-09-13T05:21:23.371088+0000","last_peered":"2024-09-13T05:21:23.371088+0000","last_clean":"2024-09-13T05:21:23.371088+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:21:23.371088+0000","last_undegraded":"2024-09-13T05:21:23.371088+0000","last_fullsized":"2024-09-13T05:21:23.371088+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'91","last_scrub_stamp":"2024-09-13T05:21:23.371025+0000","last_deep_scrub":"107'91","last_deep_scrub_stamp":"2024-09-13T05:21:23.371025+0000","last_clean_scrub_stamp":"2024-09-13T05:21:23.371025+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:40.545528+0000","scrub_duration":0.0031753549999999999,"objects_trimmed":0,"snaptrim_duration":0.00074870500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":27203,"num_write":68,"num_write_kb":49155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"107'36","reported_seq":123,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:25.022503+0000","last_change":"2024-09-13T05:21:25.022778+0000","last_active":"2024-09-13T05:21:25.022503+0000","last_peered":"2024-09-13T05:21:25.022503+0000","last_clean":"2024-09-13T05:21:25.022503+0000","last_became_active":"2024-09-13T05:20:36.954764+0000","last_became_peered":"2024-09-13T05:20:36.954764+0000","last_unstale":"2024-09-13T05:21:25.022503+0000","last_undegraded":"2024-09-13T05:21:25.022503+0000","last_fullsized":"2024-09-13T05:21:25.022503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'36","last_scrub_stamp":"2024-09-13T05:21:25.022452+0000","last_deep_scrub":"107'36","last_deep_scrub_stamp":"2024-09-13T05:21:25.022452+0000","last_clean_scrub_stamp":"2024-09-13T05:21:25.022452+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:04.392497+0000","scrub_duration":0.003446808,"objects_trimmed":0,"snaptrim_duration":0.00054390799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":8215,"num_write":27,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"121'105","reported_seq":578,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:20.751303+0000","last_change":"2024-09-13T05:21:20.751440+0000","last_active":"2024-09-13T05:21:20.751303+0000","last_peered":"2024-09-13T05:21:20.751303+0000","last_clean":"2024-09-13T05:21:20.751303+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:21:20.751303+0000","last_undegraded":"2024-09-13T05:21:20.751303+0000","last_fullsized":"2024-09-13T05:21:20.751303+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'105","last_scrub_stamp":"2024-09-13T05:21:20.751274+0000","last_deep_scrub":"121'105","last_deep_scrub_stamp":"2024-09-13T05:21:20.751274+0000","last_clean_scrub_stamp":"2024-09-13T05:21:20.751274+0000","objects_scrubbed":1,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:51:55.569402+0000","scrub_duration":0.003313248,"objects_trimmed":0,"snaptrim_duration":0.00070499600000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":38450,"num_write":79,"num_write_kb":38939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"107'97","reported_seq":415,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:23.519022+0000","last_change":"2024-09-13T05:21:23.519235+0000","last_active":"2024-09-13T05:21:23.519022+0000","last_peered":"2024-09-13T05:21:23.519022+0000","last_clean":"2024-09-13T05:21:23.519022+0000","last_became_active":"2024-09-13T05:20:36.957689+0000","last_became_peered":"2024-09-13T05:20:36.957689+0000","last_unstale":"2024-09-13T05:21:23.519022+0000","last_undegraded":"2024-09-13T05:21:23.519022+0000","last_fullsized":"2024-09-13T05:21:23.519022+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'97","last_scrub_stamp":"2024-09-13T05:21:23.518968+0000","last_deep_scrub":"107'97","last_deep_scrub_stamp":"2024-09-13T05:21:23.518968+0000","last_clean_scrub_stamp":"2024-09-13T05:21:23.518968+0000","objects_scrubbed":2,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:25:23.576920+0000","scrub_duration":0.0042684949999999998,"objects_trimmed":0,"snaptrim_duration":0.001858878,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":16522,"num_write":101,"num_write_kb":32810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":33554508,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"107'73","reported_seq":55,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:23.648570+0000","last_change":"2024-09-13T05:21:23.648805+0000","last_active":"2024-09-13T05:21:23.648570+0000","last_peered":"2024-09-13T05:21:23.648570+0000","last_clean":"2024-09-13T05:21:23.648570+0000","last_became_active":"2024-09-13T05:21:06.157892+0000","last_became_peered":"2024-09-13T05:21:06.157892+0000","last_unstale":"2024-09-13T05:21:23.648570+0000","last_undegraded":"2024-09-13T05:21:23.648570+0000","last_fullsized":"2024-09-13T05:21:23.648570+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'73","last_scrub_stamp":"2024-09-13T05:21:23.648521+0000","last_deep_scrub":"107'73","last_deep_scrub_stamp":"2024-09-13T05:21:23.648521+0000","last_clean_scrub_stamp":"2024-09-13T05:21:23.648521+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:46:46.995074+0000","scrub_duration":0.0036727309999999998,"objects_trimmed":0,"snaptrim_duration":0.00081846100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":19146,"num_write":56,"num_write_kb":41013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":41992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'84","reported_seq":223,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:24.478015+0000","last_change":"2024-09-13T05:21:24.478247+0000","last_active":"2024-09-13T05:21:24.478015+0000","last_peered":"2024-09-13T05:21:24.478015+0000","last_clean":"2024-09-13T05:21:24.478015+0000","last_became_active":"2024-09-13T05:20:43.986894+0000","last_became_peered":"2024-09-13T05:20:43.986894+0000","last_unstale":"2024-09-13T05:21:24.478015+0000","last_undegraded":"2024-09-13T05:21:24.478015+0000","last_fullsized":"2024-09-13T05:21:24.478015+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'84","last_scrub_stamp":"2024-09-13T05:21:24.477963+0000","last_deep_scrub":"107'84","last_deep_scrub_stamp":"2024-09-13T05:21:24.477963+0000","last_clean_scrub_stamp":"2024-09-13T05:21:24.477963+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:39.595228+0000","scrub_duration":0.0034909810000000002,"objects_trimmed":0,"snaptrim_duration":0.0019911320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":516,"num_read_kb":12708,"num_write":57,"num_write_kb":24588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29360244,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"107'243","reported_seq":1610,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:14.750435+0000","last_change":"2024-09-13T05:21:09.052792+0000","last_active":"2024-09-13T05:21:14.750435+0000","last_peered":"2024-09-13T05:21:14.750435+0000","last_clean":"2024-09-13T05:21:14.750435+0000","last_became_active":"2024-09-13T05:21:09.051755+0000","last_became_peered":"2024-09-13T05:21:09.051755+0000","last_unstale":"2024-09-13T05:21:14.750435+0000","last_undegraded":"2024-09-13T05:21:14.750435+0000","last_fullsized":"2024-09-13T05:21:14.750435+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T05:18:15.101421+0000","last_clean_scrub_stamp":"2024-09-13T05:18:15.101421+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291826,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1348,"num_read_kb":48395,"num_write":204,"num_write_kb":49262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":23080975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8276,"num_read_kb":199160,"num_write":984,"num_write_kb":272957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":161608112,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1128,"ondisk_log_size":1128,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607600,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":2031616,"data_stored":1153372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.071,"15min":1.071},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.70399999999999996}]},{"osd":1,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51900000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76200000000000001}]},{"osd":2,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.80800000000000005}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0580000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.88600000000000001}]},{"osd":4,"last update":"Fri Sep 13 05:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.66200000000000003}]}]},{"osd":4,"up_from":54,"seq":231928234011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.74299999999999999}]},{"osd":1,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.81799999999999995}]},{"osd":2,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.77100000000000002}]},{"osd":3,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":0.70799999999999996}]}]},{"osd":3,"up_from":29,"seq":124554051622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29508,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342332,"statfs":{"total":96636764160,"available":96606547968,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69599999999999995,"15min":0.69999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.69999999999999996}]},{"osd":1,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68300000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.67700000000000005}]},{"osd":2,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67800000000000005,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92000000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.53200000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.73099999999999998}]},{"osd":4,"last update":"Fri Sep 13 05:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.65000000000000002,"15min":0.64800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.85699999999999998,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72099999999999997,"15min":0.71499999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.63900000000000001}]},{"osd":5,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68300000000000005,"15min":0.66800000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.57499999999999996}]}]},{"osd":2,"up_from":12,"seq":51539607599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36612,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335228,"statfs":{"total":96636764160,"available":96599273472,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.28299999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.95899999999999996,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.56999999999999995}]},{"osd":1,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.59099999999999997},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.083,"5min":1.171,"15min":1.171},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86099999999999999,"5min":1.21,"15min":1.21},"last":0.64300000000000002}]},{"osd":3,"last update":"Fri Sep 13 05:20:45 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.90900000000000003,"5min":1.083,"15min":1.083},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.746,"5min":0.751,"15min":0.752},"min":{"1min":0.54700000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93700000000000006,"5min":1.21,"15min":1.21},"last":0.878}]},{"osd":4,"last update":"Fri Sep 13 05:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.72799999999999998,"15min":0.72399999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.86499999999999999,"15min":0.871},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.212,"5min":1.607,"15min":1.607},"last":0.84099999999999997}]},{"osd":5,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3799999999999999,"5min":1.544,"15min":1.544},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.95099999999999996}]}]},{"osd":1,"up_from":12,"seq":51539607599,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":2035712,"data_stored":1153391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.96999999999999997,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.625,"15min":0.627},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.85199999999999998}]},{"osd":2,"last update":"Fri Sep 13 05:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56799999999999995,"15min":0.60099999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58699999999999997,"15min":0.61499999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.755}]},{"osd":3,"last update":"Fri Sep 13 05:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72299999999999998,"15min":0.72099999999999997},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":1.117},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.98199999999999998,"5min":1.218,"15min":1.218},"last":0.94199999999999995}]},{"osd":4,"last update":"Fri Sep 13 05:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.74099999999999999,"15min":0.746},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":1.0009999999999999}]},{"osd":5,"last update":"Fri Sep 13 05:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.82699999999999996,"15min":0.85299999999999998},"min":{"1min":0.46600000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1359999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":1.048},{"interface":"front","average":{"1min":0.753,"5min":0.81999999999999995,"15min":0.85599999999999998},"min":{"1min":0.59499999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.056,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.093}]}]},{"osd":0,"up_from":12,"seq":51539607599,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1441792,"data_stored":563023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.76500000000000001}]},{"osd":3,"last update":"Fri Sep 13 05:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54800000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.872}]},{"osd":4,"last update":"Fri Sep 13 05:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.74299999999999999,"15min":0.73599999999999999},"min":{"1min":0.58199999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.026,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.88700000000000001,"15min":0.88200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2929999999999999,"5min":1.456,"15min":1.456},"last":1.022}]},{"osd":5,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.81299999999999994,"15min":0.79900000000000004},"min":{"1min":0.60699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.163,"5min":1.694,"15min":1.694},"last":0.80400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:21:43.721 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T05:18:15.101421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=18, tm_sec=15, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=5, tm_min=21, tm_sec=19, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T05:21:43.721 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T05:22:03.723 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T05:22:04.033 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-13T05:22:04.033 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-13T05:22:04.051 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-13T05:22:02.110096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8392,"num_read_kb":199260,"num_write":1143,"num_write_kb":275373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":162198480,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1247,"ondisk_log_size":1247,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":202544,"kb_used_data":9592,"kb_used_omap":14,"kb_used_meta":192881,"kb_avail":566028496,"statfs":{"total":579820584960,"available":579613179904,"internally_reserved":0,"allocated":9822208,"data_stored":4558798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15242,"internal_metadata":197510262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003718"},"pg_stats":[{"pgid":"1.0","version":"121'119","reported_seq":194,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:26.780447+0000","last_change":"2024-09-13T05:21:26.780696+0000","last_active":"2024-09-13T05:21:26.780447+0000","last_peered":"2024-09-13T05:21:26.780447+0000","last_clean":"2024-09-13T05:21:26.780447+0000","last_became_active":"2024-09-13T05:21:09.054143+0000","last_became_peered":"2024-09-13T05:21:09.054143+0000","last_unstale":"2024-09-13T05:21:26.780447+0000","last_undegraded":"2024-09-13T05:21:26.780447+0000","last_fullsized":"2024-09-13T05:21:26.780447+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'119","last_scrub_stamp":"2024-09-13T05:21:26.780387+0000","last_deep_scrub":"121'119","last_deep_scrub_stamp":"2024-09-13T05:21:26.780387+0000","last_clean_scrub_stamp":"2024-09-13T05:21:26.780387+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:27:43.835450+0000","scrub_duration":0.0095001479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"121'399","reported_seq":2754,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:35.836118+0000","last_change":"2024-09-13T05:21:35.836396+0000","last_active":"2024-09-13T05:21:35.836118+0000","last_peered":"2024-09-13T05:21:35.836118+0000","last_clean":"2024-09-13T05:21:35.836118+0000","last_became_active":"2024-09-13T05:21:09.053687+0000","last_became_peered":"2024-09-13T05:21:09.053687+0000","last_unstale":"2024-09-13T05:21:35.836118+0000","last_undegraded":"2024-09-13T05:21:35.836118+0000","last_fullsized":"2024-09-13T05:21:35.836118+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'399","last_scrub_stamp":"2024-09-13T05:21:35.836062+0000","last_deep_scrub":"121'399","last_deep_scrub_stamp":"2024-09-13T05:21:35.836062+0000","last_clean_scrub_stamp":"2024-09-13T05:21:35.836062+0000","objects_scrubbed":1,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T05:32:49.408912+0000","scrub_duration":0.0064857179999999997,"objects_trimmed":0,"snaptrim_duration":0.00070414999999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5324,"num_read_kb":28521,"num_write":392,"num_write_kb":20803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":12648486,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"107'91","reported_seq":448,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:23.371088+0000","last_change":"2024-09-13T05:21:23.371366+0000","last_active":"2024-09-13T05:21:23.371088+0000","last_peered":"2024-09-13T05:21:23.371088+0000","last_clean":"2024-09-13T05:21:23.371088+0000","last_became_active":"2024-09-13T05:18:16.127958+0000","last_became_peered":"2024-09-13T05:18:16.127958+0000","last_unstale":"2024-09-13T05:21:23.371088+0000","last_undegraded":"2024-09-13T05:21:23.371088+0000","last_fullsized":"2024-09-13T05:21:23.371088+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'91","last_scrub_stamp":"2024-09-13T05:21:23.371025+0000","last_deep_scrub":"107'91","last_deep_scrub_stamp":"2024-09-13T05:21:23.371025+0000","last_clean_scrub_stamp":"2024-09-13T05:21:23.371025+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:40.545528+0000","scrub_duration":0.0031753549999999999,"objects_trimmed":0,"snaptrim_duration":0.00074870500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":214,"num_read_kb":27203,"num_write":68,"num_write_kb":49155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"107'36","reported_seq":123,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:25.022503+0000","last_change":"2024-09-13T05:21:25.022778+0000","last_active":"2024-09-13T05:21:25.022503+0000","last_peered":"2024-09-13T05:21:25.022503+0000","last_clean":"2024-09-13T05:21:25.022503+0000","last_became_active":"2024-09-13T05:20:36.954764+0000","last_became_peered":"2024-09-13T05:20:36.954764+0000","last_unstale":"2024-09-13T05:21:25.022503+0000","last_undegraded":"2024-09-13T05:21:25.022503+0000","last_fullsized":"2024-09-13T05:21:25.022503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'36","last_scrub_stamp":"2024-09-13T05:21:25.022452+0000","last_deep_scrub":"107'36","last_deep_scrub_stamp":"2024-09-13T05:21:25.022452+0000","last_clean_scrub_stamp":"2024-09-13T05:21:25.022452+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:16:04.392497+0000","scrub_duration":0.003446808,"objects_trimmed":0,"snaptrim_duration":0.00054390799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":8215,"num_write":27,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8388662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"121'105","reported_seq":578,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:20.751303+0000","last_change":"2024-09-13T05:21:20.751440+0000","last_active":"2024-09-13T05:21:20.751303+0000","last_peered":"2024-09-13T05:21:20.751303+0000","last_clean":"2024-09-13T05:21:20.751303+0000","last_became_active":"2024-09-13T05:19:24.467787+0000","last_became_peered":"2024-09-13T05:19:24.467787+0000","last_unstale":"2024-09-13T05:21:20.751303+0000","last_undegraded":"2024-09-13T05:21:20.751303+0000","last_fullsized":"2024-09-13T05:21:20.751303+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'105","last_scrub_stamp":"2024-09-13T05:21:20.751274+0000","last_deep_scrub":"121'105","last_deep_scrub_stamp":"2024-09-13T05:21:20.751274+0000","last_clean_scrub_stamp":"2024-09-13T05:21:20.751274+0000","objects_scrubbed":1,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:51:55.569402+0000","scrub_duration":0.003313248,"objects_trimmed":0,"snaptrim_duration":0.00070499600000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":38450,"num_write":79,"num_write_kb":38939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"107'97","reported_seq":415,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:23.519022+0000","last_change":"2024-09-13T05:21:23.519235+0000","last_active":"2024-09-13T05:21:23.519022+0000","last_peered":"2024-09-13T05:21:23.519022+0000","last_clean":"2024-09-13T05:21:23.519022+0000","last_became_active":"2024-09-13T05:20:36.957689+0000","last_became_peered":"2024-09-13T05:20:36.957689+0000","last_unstale":"2024-09-13T05:21:23.519022+0000","last_undegraded":"2024-09-13T05:21:23.519022+0000","last_fullsized":"2024-09-13T05:21:23.519022+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'97","last_scrub_stamp":"2024-09-13T05:21:23.518968+0000","last_deep_scrub":"107'97","last_deep_scrub_stamp":"2024-09-13T05:21:23.518968+0000","last_clean_scrub_stamp":"2024-09-13T05:21:23.518968+0000","objects_scrubbed":2,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:25:23.576920+0000","scrub_duration":0.0042684949999999998,"objects_trimmed":0,"snaptrim_duration":0.001858878,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":16522,"num_write":101,"num_write_kb":32810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":33554508,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"107'73","reported_seq":55,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:23.648570+0000","last_change":"2024-09-13T05:21:23.648805+0000","last_active":"2024-09-13T05:21:23.648570+0000","last_peered":"2024-09-13T05:21:23.648570+0000","last_clean":"2024-09-13T05:21:23.648570+0000","last_became_active":"2024-09-13T05:21:06.157892+0000","last_became_peered":"2024-09-13T05:21:06.157892+0000","last_unstale":"2024-09-13T05:21:23.648570+0000","last_undegraded":"2024-09-13T05:21:23.648570+0000","last_fullsized":"2024-09-13T05:21:23.648570+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'73","last_scrub_stamp":"2024-09-13T05:21:23.648521+0000","last_deep_scrub":"107'73","last_deep_scrub_stamp":"2024-09-13T05:21:23.648521+0000","last_clean_scrub_stamp":"2024-09-13T05:21:23.648521+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:46:46.995074+0000","scrub_duration":0.0036727309999999998,"objects_trimmed":0,"snaptrim_duration":0.00081846100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":229,"num_read_kb":19146,"num_write":56,"num_write_kb":41013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":41992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"107'84","reported_seq":223,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:24.478015+0000","last_change":"2024-09-13T05:21:24.478247+0000","last_active":"2024-09-13T05:21:24.478015+0000","last_peered":"2024-09-13T05:21:24.478015+0000","last_clean":"2024-09-13T05:21:24.478015+0000","last_became_active":"2024-09-13T05:20:43.986894+0000","last_became_peered":"2024-09-13T05:20:43.986894+0000","last_unstale":"2024-09-13T05:21:24.478015+0000","last_undegraded":"2024-09-13T05:21:24.478015+0000","last_fullsized":"2024-09-13T05:21:24.478015+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'84","last_scrub_stamp":"2024-09-13T05:21:24.477963+0000","last_deep_scrub":"107'84","last_deep_scrub_stamp":"2024-09-13T05:21:24.477963+0000","last_clean_scrub_stamp":"2024-09-13T05:21:24.477963+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:39.595228+0000","scrub_duration":0.0034909810000000002,"objects_trimmed":0,"snaptrim_duration":0.0019911320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":516,"num_read_kb":12708,"num_write":57,"num_write_kb":24588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29360244,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"107'243","reported_seq":1621,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-13T05:21:38.777077+0000","last_change":"2024-09-13T05:21:38.777317+0000","last_active":"2024-09-13T05:21:38.777077+0000","last_peered":"2024-09-13T05:21:38.777077+0000","last_clean":"2024-09-13T05:21:38.777077+0000","last_became_active":"2024-09-13T05:21:09.051755+0000","last_became_peered":"2024-09-13T05:21:09.051755+0000","last_unstale":"2024-09-13T05:21:38.777077+0000","last_undegraded":"2024-09-13T05:21:38.777077+0000","last_fullsized":"2024-09-13T05:21:38.777077+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'243","last_scrub_stamp":"2024-09-13T05:21:38.777021+0000","last_deep_scrub":"107'243","last_deep_scrub_stamp":"2024-09-13T05:21:38.777021+0000","last_clean_scrub_stamp":"2024-09-13T05:21:38.777021+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:25.684175+0000","scrub_duration":0.0037755240000000002,"objects_trimmed":0,"snaptrim_duration":0.001291826,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1348,"num_read_kb":48395,"num_write":204,"num_write_kb":49262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":23080975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8276,"num_read_kb":199160,"num_write":984,"num_write_kb":272957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":161608112,"num_keys_recovered":41,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1128,"ondisk_log_size":1128,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607604,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":2031616,"data_stored":1153372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.071,"15min":1.071},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95399999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.72599999999999998}]},{"osd":1,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.53700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.51900000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.75}]},{"osd":2,"last update":"Fri Sep 13 05:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.72299999999999998},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Sep 13 05:21:50 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.68700000000000006,"15min":0.68300000000000005},"min":{"1min":0.52500000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.96699999999999997,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.67800000000000005}]},{"osd":4,"last update":"Fri Sep 13 05:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.45700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.501,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.997,"5min":1.125,"15min":1.125},"last":0.877}]}]},{"osd":4,"up_from":54,"seq":231928234015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29516,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342324,"statfs":{"total":96636764160,"available":96606539776,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4437,"internal_metadata":28765867},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.67000000000000004,"15min":0.65200000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.875}]},{"osd":1,"last update":"Fri Sep 13 05:21:41 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.70299999999999996,"15min":0.69999999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.86599999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.71199999999999997,"15min":0.69599999999999995},"min":{"1min":0.57199999999999995,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.77600000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:21:41 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.78600000000000003,"15min":0.79200000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91200000000000003,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.73799999999999999,"15min":0.73299999999999998},"min":{"1min":0.55300000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.72699999999999998}]},{"osd":3,"last update":"Fri Sep 13 05:21:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.68400000000000005,"15min":0.68600000000000005},"min":{"1min":0.40100000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.92300000000000004,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.76800000000000002,"15min":0.77500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90100000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.59599999999999997}]},{"osd":5,"last update":"Fri Sep 13 05:21:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.94199999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.82299999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.97599999999999998,"5min":2.153,"15min":2.153},"last":0.83299999999999996}]}]},{"osd":3,"up_from":29,"seq":124554051626,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29508,"kb_used_data":1404,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342332,"statfs":{"total":96636764160,"available":96606547968,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4429,"internal_metadata":28765875},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.69999999999999996,"15min":0.68200000000000005},"min":{"1min":0.55200000000000005,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.72699999999999998,"15min":0.70999999999999996},"min":{"1min":0.69199999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.222,"5min":1.252,"15min":1.252},"last":0.91900000000000004}]},{"osd":1,"last update":"Fri Sep 13 05:21:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.72199999999999998,"15min":0.70199999999999996},"min":{"1min":0.69299999999999995,"5min":0.314,"15min":0.314},"max":{"1min":0.96799999999999997,"5min":1.028,"15min":1.028},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.69099999999999995,"15min":0.68400000000000005},"min":{"1min":0.56299999999999994,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84699999999999998,"5min":1.125,"15min":1.125},"last":0.89800000000000002}]},{"osd":2,"last update":"Fri Sep 13 05:21:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.69999999999999996,"15min":0.68600000000000005},"min":{"1min":0.62,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.505,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0329999999999999,"5min":1.165,"15min":1.165},"last":1.077}]},{"osd":4,"last update":"Fri Sep 13 05:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.65000000000000002,"15min":0.64800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.85699999999999998,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72099999999999997,"15min":0.71499999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.753}]},{"osd":5,"last update":"Fri Sep 13 05:21:49 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.71199999999999997,"15min":0.67800000000000005},"min":{"1min":0.60099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.1830000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.65500000000000003,"15min":0.63300000000000001},"min":{"1min":0.55800000000000005,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93500000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.83699999999999997}]}]},{"osd":2,"up_from":12,"seq":51539607603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36612,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335228,"statfs":{"total":96636764160,"available":96599273472,"internally_reserved":0,"allocated":1437696,"data_stored":563004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.28299999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.95899999999999996,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.81200000000000006}]},{"osd":1,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.59099999999999997},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.083,"5min":1.171,"15min":1.171},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86099999999999999,"5min":1.21,"15min":1.21},"last":0.64100000000000001}]},{"osd":3,"last update":"Fri Sep 13 05:21:47 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.71699999999999997,"15min":0.69099999999999995},"min":{"1min":0.44800000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.76100000000000001,"15min":0.755},"min":{"1min":0.52100000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.72199999999999998}]},{"osd":4,"last update":"Fri Sep 13 05:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.72799999999999998,"15min":0.72399999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.86499999999999999,"15min":0.871},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.212,"5min":1.607,"15min":1.607},"last":0.93200000000000005}]},{"osd":5,"last update":"Fri Sep 13 05:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3799999999999999,"5min":1.544,"15min":1.544},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1399999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.78100000000000003}]}]},{"osd":1,"up_from":12,"seq":51539607603,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37196,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334644,"statfs":{"total":96636764160,"available":96598675456,"internally_reserved":0,"allocated":2035712,"data_stored":1153391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 05:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.96999999999999997,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.625,"15min":0.627},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":1.0029999999999999}]},{"osd":2,"last update":"Fri Sep 13 05:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56799999999999995,"15min":0.60099999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58699999999999997,"15min":0.61499999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Sep 13 05:21:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.73899999999999999,"15min":0.72699999999999998},"min":{"1min":0.52300000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.77600000000000002,"15min":0.78100000000000003},"min":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0209999999999999,"5min":1.218,"15min":1.218},"last":0.85099999999999998}]},{"osd":4,"last update":"Fri Sep 13 05:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.74099999999999999,"15min":0.746},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.94699999999999995}]},{"osd":5,"last update":"Fri Sep 13 05:21:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.82699999999999996,"15min":0.85299999999999998},"min":{"1min":0.46600000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1359999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.81999999999999995,"15min":0.85599999999999998},"min":{"1min":0.59499999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.056,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.069}]}]},{"osd":0,"up_from":12,"seq":51539607603,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1441792,"data_stored":563023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.82799999999999996}]},{"osd":3,"last update":"Fri Sep 13 05:21:50 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.78700000000000003,"15min":0.75900000000000001},"min":{"1min":0.68400000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.82599999999999996,"15min":0.79700000000000004},"min":{"1min":0.47599999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3520000000000001,"5min":1.361,"15min":1.361},"last":1.02}]},{"osd":4,"last update":"Fri Sep 13 05:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.74299999999999999,"15min":0.73599999999999999},"min":{"1min":0.58199999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.026,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.88700000000000001,"15min":0.88200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2929999999999999,"5min":1.456,"15min":1.456},"last":0.97299999999999998}]},{"osd":5,"last update":"Fri Sep 13 05:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.81299999999999994,"15min":0.79900000000000004},"min":{"1min":0.60699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.60699999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.163,"5min":1.694,"15min":1.694},"last":0.79200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T05:22:04.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T05:22:04.382 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T05:22:04.382 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T05:22:04.382 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T05:22:04.382 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:04.472 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T05:22:04.472 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T05:22:04.472 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:04.571 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T05:22:04.571 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T05:22:04.572 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:07.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-13T05:22:10.673 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T05:22:10.674 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T05:22:10.674 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:10.743 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T05:22:10.743 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T05:22:10.743 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:10.823 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T05:22:10.823 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T05:22:10.823 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:10.915 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T05:22:10.915 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T05:22:10.916 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T05:22:10.916 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:10.946 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T05:22:10.946 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T05:22:10.946 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T05:22:10.946 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:10.966 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T05:22:10.966 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T05:22:10.966 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:11.028 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T05:22:11.028 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T05:22:11.029 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T05:22:11.042 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T05:22:11.042 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T05:22:11.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | head -n 1 2024-09-13T05:22:11.086 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-13T05:22:11.087 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T05:22:11.390 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-13T05:22:11.390 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T05:22:11.687 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-13T05:22:11.688 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T05:22:11.863 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:22:11.863 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T05:22:12.064 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:22:12.064 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T05:22:12.268 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:22:12.269 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T05:22:12.461 INFO:tasks.ceph:Archiving mon data... 2024-09-13T05:22:12.462 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/data/mon.a.tgz 2024-09-13T05:22:12.463 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-13T05:22:12.469 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.1iWJvqt0Uy 2024-09-13T05:22:12.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.1iWJvqt0Uy 2024-09-13T05:22:12.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.1iWJvqt0Uy 2024-09-13T05:22:12.610 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-13T05:22:12.658 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.1iWJvqt0Uy is 297KB 2024-09-13T05:22:12.720 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.1iWJvqt0Uy 2024-09-13T05:22:12.725 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/data/mon.c.tgz 2024-09-13T05:22:12.725 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-13T05:22:12.773 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.dVMKqmnTox 2024-09-13T05:22:12.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.dVMKqmnTox 2024-09-13T05:22:12.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.dVMKqmnTox 2024-09-13T05:22:12.986 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.dVMKqmnTox is 413KB 2024-09-13T05:22:13.047 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.dVMKqmnTox 2024-09-13T05:22:13.053 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/data/mon.b.tgz 2024-09-13T05:22:13.053 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-09-13T05:22:13.058 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.1diF1PvGrA 2024-09-13T05:22:13.059 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.1diF1PvGrA 2024-09-13T05:22:13.208 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.1diF1PvGrA 2024-09-13T05:22:13.271 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.1diF1PvGrA is 413KB 2024-09-13T05:22:13.334 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.1diF1PvGrA 2024-09-13T05:22:13.340 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T05:22:13.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T05:22:13.342 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T05:22:13.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T05:22:13.400 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-13T05:22:13.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T05:22:13.402 DEBUG:teuthology.orchestra.run.smithi079:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T05:22:13.427 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T05:22:13.441 DEBUG:teuthology.orchestra.run.smithi122:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T05:22:13.466 INFO:teuthology.orchestra.run.smithi079.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T05:22:13.471 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T05:22:13.474 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T05:22:13.476 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/remote/smithi016/crash 2024-09-13T05:22:13.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T05:22:13.490 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/remote/smithi079/crash 2024-09-13T05:22:13.491 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T05:22:13.523 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/remote/smithi122/crash 2024-09-13T05:22:13.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T05:22:13.536 INFO:tasks.ceph:Compressing logs... 2024-09-13T05:22:13.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T05:22:13.539 DEBUG:teuthology.orchestra.run.smithi079:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T05:22:13.569 DEBUG:teuthology.orchestra.run.smithi122:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T05:22:15.107 INFO:tasks.ceph:Archiving logs... 2024-09-13T05:22:15.107 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/remote/smithi016/log 2024-09-13T05:22:15.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T05:22:15.463 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/remote/smithi079/log 2024-09-13T05:22:15.464 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T05:22:15.569 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/remote/smithi122/log 2024-09-13T05:22:15.570 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T05:22:15.780 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T05:22:15.793 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T05:22:15.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T05:22:15.796 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T05:22:15.798 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T05:22:15.914 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T05:22:15.933 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T05:22:15.933 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T05:22:15.936 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T05:22:15.938 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T05:22:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T05:22:15.946 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== 2024-09-13T05:22:15.946 INFO:teuthology.orchestra.run.smithi016.stdout:+hv01.front.sepi 67.205.162.81 3 u 62 64 77 0.094 -1.079 0.696 2024-09-13T05:22:15.947 INFO:teuthology.orchestra.run.smithi016.stdout:+hv02.front.sepi 63.231.80.2 3 u 62 64 77 0.093 +1.014 0.696 2024-09-13T05:22:15.947 INFO:teuthology.orchestra.run.smithi016.stdout:*hv03.front.sepi 74.6.168.72 3 u 65 64 77 0.096 -0.494 0.407 2024-09-13T05:22:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:22:15.950 INFO:teuthology.orchestra.run.smithi079.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T05:22:15.950 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================== 2024-09-13T05:22:15.950 INFO:teuthology.orchestra.run.smithi079.stdout:+hv01.front.sepi 67.205.162.81 3 u 62 64 77 0.125 -1.016 1.513 2024-09-13T05:22:15.950 INFO:teuthology.orchestra.run.smithi079.stdout:+hv02.front.sepi 63.231.80.2 3 u 1 64 177 0.141 -0.219 1.310 2024-09-13T05:22:15.950 INFO:teuthology.orchestra.run.smithi079.stdout:*hv03.front.sepi 74.6.168.72 3 u - 64 177 0.129 -1.364 1.267 2024-09-13T05:22:15.950 INFO:teuthology.orchestra.run.smithi079.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:22:15.952 INFO:teuthology.orchestra.run.smithi122.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T05:22:15.952 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================== 2024-09-13T05:22:15.952 INFO:teuthology.orchestra.run.smithi122.stdout:+hv01.front.sepi 67.205.162.81 3 u 62 64 77 0.087 -0.427 1.864 2024-09-13T05:22:15.952 INFO:teuthology.orchestra.run.smithi122.stdout:+hv02.front.sepi 63.231.80.2 3 u 63 64 77 0.139 +0.593 1.937 2024-09-13T05:22:15.952 INFO:teuthology.orchestra.run.smithi122.stdout:*hv03.front.sepi 74.6.168.72 3 u 1 64 177 0.110 -0.900 1.530 2024-09-13T05:22:15.952 INFO:teuthology.orchestra.run.smithi122.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T05:22:15.952 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T05:22:15.962 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T05:22:15.963 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T05:22:15.972 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T05:22:15.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T05:22:15.999 INFO:teuthology.task.internal:Duration was 861.632210 seconds 2024-09-13T05:22:15.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T05:22:16.008 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T05:22:16.008 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T05:22:16.011 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T05:22:16.013 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T05:22:16.089 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T05:22:16.089 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-09-13T05:22:16.089 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T05:22:16.099 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-09-13T05:22:16.099 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T05:22:16.109 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T05:22:16.109 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T05:22:16.120 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T05:22:16.120 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T05:22:16.142 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T05:22:16.153 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T05:22:16.192 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T05:22:16.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T05:22:16.214 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T05:22:16.221 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T05:22:16.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T05:22:16.299 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T05:22:16.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T05:22:16.312 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T05:22:16.325 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T05:22:16.335 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T05:22:16.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T05:22:16.355 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T05:22:16.362 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-09-13T05:22:16.369 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T05:22:16.379 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-09-13T05:22:16.387 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-09-13T05:22:16.401 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T05:22:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:22:16.434 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T05:22:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:22:16.448 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T05:22:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T05:22:16.452 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T05:22:16.464 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T05:22:16.464 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/remote/smithi016 2024-09-13T05:22:16.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T05:22:16.498 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/remote/smithi079 2024-09-13T05:22:16.499 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T05:22:16.521 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900857/remote/smithi122 2024-09-13T05:22:16.521 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T05:22:16.542 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T05:22:16.542 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T05:22:16.544 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T05:22:16.561 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T05:22:16.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T05:22:16.595 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T05:22:16.596 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T05:22:16.605 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T05:22:16.605 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T05:22:16.608 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T05:22:16.610 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T05:22:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: 658460 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 05:22 /home/ubuntu/cephtest 2024-09-13T05:22:16.614 INFO:teuthology.orchestra.run.smithi079.stdout: 658712 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 05:22 /home/ubuntu/cephtest 2024-09-13T05:22:16.634 INFO:teuthology.orchestra.run.smithi122.stdout: 658622 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 05:22 /home/ubuntu/cephtest 2024-09-13T05:22:16.638 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T05:22:16.648 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T05:22:16.672 INFO:teuthology.run:Summary data: description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_nbd} duration: 861.632209777832 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-13T05:22:16.673 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T05:22:16.756 INFO:teuthology.run:pass